Skip to contentSkip to conversation

-mm -> 2.6.13 merge status

DISCUSSION Andrew Morton 108 participants 526 messages Jun 21, 2005 – Jul 14, 2005 3 messages not in the archive VFS Page 1 of 3

lore
Share & tools
lore
#0 lore raw

This summarises my current thinking on various patches which are presently in -mm. I cover large things and small-but-controversial things. Anything which isn't covered here (and that's a lot of material) is probably a "will merge", unless it obviously isn't.

(If you reply to this email it would be a good idea to alter the Subject:
to reflect which feature you are discussing)

git-ocfs

The OCFS2 filesystem. OK by me, although I'm not sure it's had enough review.

sparsemem

OK by me for a merge. Need to poke arch maintainers first, check that they've looked at it sufficiently closely.

vm-early-zone-reclaim

Needs some convincing benchmark numbers to back it up. Otherwise OK.

avoiding-mmap-fragmentation

Tricky. Addresses vm area fragmentation issues due to recent optimisations to the free-area lookup code. Will merge.

periodically-drain-non-local-pagesets

Will merge

pcibus_to_node and users

Will merge

CONFIG_HZ for x86 and ia64: changes default HZ to 250, make HZ Kconfigurable.

Will merge (will switch default to 1000 Hz later if that seems necessary)

dmi-*.patch

Will merge. I have a comment "The below break x440". Maybe it got fixed. We'll doubtless hear if not.

xen-*.patch

These are little cleanups and abstractions which make a Xen merge easier. May as well merge them.

CPU hotplug for x86 and x86_64

Not really useful on current hardware, but these provide infrastructure which some power management patches need, and it seems sensible to make the reference architecture support hotplug. Will merge.

swsusp-on-SMP

Will merge.

cfq version 3

Not sure. Jens seems to be setting up a few git trees. On hold.

RCUification of the key management code

Don't know - dhowells seemed diffident last time we discussed this.

timers-fixes-improvements.patch

SMP speedups for the core timer code. It was bumpy, but this seems stable now. Will merge.

kprobes-*

Will merge

rapidio-*

Will merge.

namespace*.patch

Awaiting viro ack.

xtensa architecture

Is xtensa now, or will it be in the future a sufficiently popular architecture to justify the cost of having this code in the tree?

Heaven knows. Will merge.

dlm-*.patch: Red Hat distributed lock manager

Hard. Right now it seems that no in-kernel projects will use this and only one out-of-kernel project will use it. Shelve the problem until after Kernel Summit, where some light may be shed.

Opinions are sought...

connector.patch

Nice idea IMO, but there are still questions around the implementation. More dialogue needed ;)

connector-add-a-fork-connector.patch

OK, but needs connector.

inotify

There are still concerns about the userspace API and internal implementation details. More slogging needed.

pcmcia-*.patch

Makes the pcmcia layer generate hotplug events and deprecates cardmgr. Will merge.

NUMA-aware slab allocator

Seems stable now, but it needs some ifdef reduction work before merging, please.

CPU scheduler

Will merge some of these patches. We're still discussing which ones.

perfctr

Not yet, but getting closer. The PPC64 guys still need to sort out a few interface issues with Mikael. We might be able to fit this into 2.6.13 if people get a move on.

cachefs

This is a ton of code which knows rather a lot about pagecache internals. It allows the AFS client to cache file contents on a local blockdev.

I don't think it's a justified addition for only AFS and I'd prefer to see it proven for NFS as well.

Issues around add-page-becoming-writable-notification.patch need to be resolved.

cachefs-for-nfs

A recent addition. Needs review from NFS developers and considerably more testing.

These things aren't looking likely for 2.6.13.

kexec and kdump

I guess we should merge these.

I'm still concerned that the various device shutdown problems will mean that the success rate for crashing kernels is not high enough for kdump to be considered a success. In which case in six months time we'll hear rumours about vendors shipping wholly different crashdump implementations, which would be quite bad.

But I think this has gone as far as it can go in -mm, so it's a bit of a punt.

reiser4

Merge it, I guess.

The patches still contain all the reiser4-specific namespace enhancements, only it is disabled, so it is effectively dead code. Maybe we should ask that it actually be removed?

v9fs

I'm not sure that this has a sufficiently high usefulness-to-maintenance-cost ratio.

fuse

This is useful, but there are, AFAIK, two issues:

- We're still deadlocked over some permission-checking hacks in there

- It has an NFS server implementation which only works if the to-be-served file happens to be in dcache.

It has been said that a userspace NFS server can be used to get full NFS server functionality with FUSE. I think the half-assed kernel implementation should be done away with.

execute-in-place

Will merge. Have the embedded guys commented on the usefulness of this for execute-out-of-ROM?

#1 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 5 lines

fuse

This is useful, but there are, AFAIK, two issues:

- We're still deadlocked over some permission-checking hacks in there

Oh, god. Let me try to explain this again:

- This is a security issue with unprivileged mounts

- Since no other filesystem currently offers secure unpivileged mounts in Linux, this is something "new"

- Since it's something new, there's a big resistance to acceptance. I understand this, I only ask people, to please read Documentation/filesystems/fuse.txt, before arguing against it

- IMO it's not a hack, and it's not something that can be solved otherwise (no, private namespaces are NOT a solution, they are mosty orthogonal to this).

So I welcome constructive discussion. However bear in mind, that I definitely don't want to disable unprivileged mounts. For me that is _the_ most important feature of FUSE.

Quoted 2 lines

- It has an NFS server implementation which only works if the to-be-served file happens to be in dcache.

More preciesly it relies on icache.

Quoted 3 lines

It has been said that a userspace NFS server can be used to get full NFS server functionality with FUSE. I think the half-assed kernel implementation should be done away with.

I won't shed many tears if you drop fuse-nfs-export.patch. It would at least give the userspace solution some boost.

However the patch is pretty small, and despite it's flaws, I know it's used by a number of people.

Thanks,
Miklos

#2 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 3 lines · Miklos Szeredi

So I welcome constructive discussion. However bear in mind, that I definitely don't want to disable unprivileged mounts. For me that is _the_ most important feature of FUSE.

If the choice was "merge FUSE without unpriv mounts for now" or "discard fuse completely" which is preferable.

It seems to me (just IMHO) that it would be better to merge FUSE without that feature and then spend the time getting that feature right _in parallel_ with people using, breaking and reviewing FUSE a lot more.

#3 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 6 lines
Quoted 3 lines

So I welcome constructive discussion. However bear in mind, that I definitely don't want to disable unprivileged mounts. For me that is _the_ most important feature of FUSE.

If the choice was "merge FUSE without unpriv mounts for now" or "discard fuse completely" which is preferable.

FUSE is doing fine outside mainline, so discard wouldn't be such a big setback. Including it without unpriv mounts would effectively fork FUSE into an out-of-tree and an in-tree version, which is sure to cause confusion.

So yes, I'd prefer not merging to merging without unpriv mounts. But it's GPL, so obviously I don't have any legal control over it.

Quoted 3 lines

It seems to me (just IMHO) that it would be better to merge FUSE without that feature and then spend the time getting that feature right _in parallel_ with people using, breaking and reviewing FUSE a lot more.

The security measure in question is actually very simple (10 lines or so). So it's not the implementation that people have problems with but the concept. The concept itself is hard to swallow, because it does something unexpected, but what it does is in fact very logical.

That's why I ask people to read the documentation, think about it and _then_ argue. Up till now the discussion with Christoph Hellwig about this hasn't been on the level of rational arguments (and he's the only definite naysayer).

Thanks,
Miklos

#4 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

In reply to Miklos Szeredi

Hi!

Quoted 7 lines
Quoted 3 lines

This is useful, but there are, AFAIK, two issues:

- We're still deadlocked over some permission-checking hacks in there

Oh, god. Let me try to explain this again:

- This is a security issue with unprivileged mounts

Pretty please, just merge it without unpriviledged mounts. I see they are usefull,
but they are too strange for now. If user tries mounting themselves, he gets -EPERM, and applies 10-liner from you. Does not look like "fork" or anything serious.
Pavel

Signature · 1 line
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         
#5 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

I'd like to see FUSE merged too, even without user mounts, if only to get more motivated to actually play with this and see how it works.

John

#6 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 2 lines

I'd like to see FUSE merged too, even without user mounts, if only to get more motivated to actually play with this and see how it works.

You can try it out now. Download from fuse.sf.net, ./configure; make; make install. It's not as if it was harder than compiling a kernel :)

Miklos

#7 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

In reply to Pavel Machek
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines

This is useful, but there are, AFAIK, two issues:

- We're still deadlocked over some permission-checking hacks in there

Oh, god. Let me try to explain this again:

- This is a security issue with unprivileged mounts

Pretty please, just merge it without unpriviledged mounts. I see they are usefull, but they are too strange for now.

An emotional argument again. What's "strange" about it?

You have a choice of:

1) believe me that the current solution is fine

2) get down and try to understand the damn thing, and then come up with technical arguments for/against it

I know that 2) takes time and energy, and not a lot of people are interested enough to go through it, but why on earth do you think it will _ever_ be easier than now.

Thanks,
Miklos

#8 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Hi!

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

This is useful, but there are, AFAIK, two issues:

- We're still deadlocked over some permission-checking hacks in there

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Oh, god. Let me try to explain this again:

- This is a security issue with unprivileged mounts

Pretty please, just merge it without unpriviledged mounts. I see they are usefull, but they are too strange for now.

An emotional argument again. What's "strange" about it?

Not so emotional argument...

System where users can mount their own filesystems should not be called "Unix" any more. It introduces new mechanism, similar to ptrace. It restricts root in ways not seen before. How is updatedb/locate supposed to work on system with this? How is it going to interact with backup tools?

Add this to your A): "by tricking some interpretter to think script is setuid".

Quoted 5 lines

You have a choice of: 1) believe me that the current solution is fine

2) get down and try to understand the damn thing, and then come up with technical arguments for/against it

Argument is "it is **** ugly".

Your fuse.txt explains why it is not security hole. It does not explain why your interface is the best possible, and what alternative ways of "not security hole" exist.

Pavel

Signature · 1 line
teflon -- maybe it is a trademark, but it should not be.
#9 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 4 lines

Not so emotional argument...

System where users can mount their own filesystems should not be called "Unix" any more.

It's not. It's "Linux". And anyway, sysadmin may set whatever owner/group/permissions on '/dev/fuse' to disallow or selectively allow users to be able to mount FUSE filesystems.

Quoted 2 lines

It introduces new mechanism, similar to ptrace. It restricts root in ways not seen before.

Not true. Root squash in NFS has similar effect.

Quoted 2 lines

How is updatedb/locate supposed to work on system with this? How is it going to interact with backup tools?

I assure you, that it will cause no problems whatever. These programs are able to gracefully handle errors.

Quoted 2 lines

Add this to your A): "by tricking some interpretter to think script is setuid".

How would you do that?

Quoted 7 lines
Quoted 5 lines

You have a choice of: 1) believe me that the current solution is fine

2) get down and try to understand the damn thing, and then come up with technical arguments for/against it

Argument is "it is **** ugly".

Yeah, that's your opinion. Mine is that it's f****** beautiful ;).

There are plenty of ugly things in Unix/Linux that you've become so accustomed to, that they no longer seem ugly. Think about the sticky bit on directories for example. That one was breaking assumptions left and right when it got introduced, but people came to accept it, because it's useful.

Quoted 3 lines

Your fuse.txt explains why it is not security hole. It does not explain why your interface is the best possible, and what alternative ways of "not security hole" exist.

That's because I don't see any alternative. The "preventing user from tracing root" and "preventing access to user's filesysem by root" must come together. There's doesn't seem to be any other way.

BTW, thanks for reading through fuse.txt :)

Miklos

#10 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 7 lines · Miklos Szeredi
Quoted 5 lines

Not so emotional argument...

System where users can mount their own filesystems should not be called "Unix" any more.

It's not. It's "Linux".

It would be helpful if we could have a brief description of the feature which you're discussing here. We discussed this a couple of months back, but I've forgotten most of it and it was off-list I think.

Doing `grep uid fs/fuse/*.c' gets us to the implementation, yes?

Which parts are controversial?

How _should_ we implement unprivileged mounts, if not this way?

#11 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 7 lines

It would be helpful if we could have a brief description of the feature which you're discussing here. We discussed this a couple of months back, but I've forgotten most of it and it was off-list I think.

Doing `grep uid fs/fuse/*.c' gets us to the implementation, yes?

Which parts are controversial?

The controversial part is fuse_allow_task() called from fuse_permission() and fuse_revalidate() (fs/fuse/dir.c).

This function (as explained by the header comment) disallows access to the filesystem for any task, which the filesystem owner (the user who did the mount) is not allowed to ptrace.

The rationale is that accessing the filesystem gives the filesystem implementor ptrace like capabilities (detailed in Documentation/filesystems/fuse.txt)

It is controversial, because obviously root owned tasks are not ptrace-able by the user, and so these tasks will be denied access to the user mounted filesystem (-EACCESS is returned on stat() or any other file operation).

However nobody raised _any_ concrete technical problem associated with this, and the 4 years of widespread use didn't turn up any either. So IMO it's "ugly" only in people's heads and not in reality.

Quoted 1 line

How _should_ we implement unprivileged mounts, if not this way?

A lot of the earlier discussion centered around making private namespaces usable, and so segregating user mounts. However because suid/sgid programs can still gain elevated privileges in a private namespace the above check is still necessary.

So private namespaces are alone not a solution.

I also suggested (and implemented a patch) which hides the user mounts, based on the above check. This would mean, that root and other users would simply not see the user's mount instead of getting an error. This was also received with very little enthusiasm.

A further suggested alternative was to confine user mounts to a single directory (e.g. '/mnt/usermounts'). This would solve the problem if every daemon and suid/sgid program was tought to not enter this directory. This is quite impractical. Besides this is a very inflexible solution.

I'm still waiting for a practical alternative. I don't consider practical the opinion held by quite a few people:

"let's integrate FUSE now, without unpriv mounts, maybe we'll find
a better solution later"

I consider it highly unlikely, that any time will be better than now.

Thanks,
Miklos

#12 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 7 lines · Miklos Szeredi
Quoted 7 lines

It would be helpful if we could have a brief description of the feature which you're discussing here. We discussed this a couple of months back, but I've forgotten most of it and it was off-list I think.

Doing `grep uid fs/fuse/*.c' gets us to the implementation, yes?

Quoted 8 lines
Quoted 1 line

Which parts are controversial?

The controversial part is fuse_allow_task() called from fuse_permission() and fuse_revalidate() (fs/fuse/dir.c).

This function (as explained by the header comment) disallows access to the filesystem for any task, which the filesystem owner (the user who did the mount) is not allowed to ptrace.

That's fairly weird. Overloading ptraceability is awkward, but also the *direction* is wrong. It's saying "if I can ptrace you, you can read my data". I'd have expected to see "if you can ptrace me, you can access my data".

Quoted 3 lines

The rationale is that accessing the filesystem gives the filesystem implementor ptrace like capabilities (detailed in Documentation/filesystems/fuse.txt)

hrm. Makes some sense.

Quoted 8 lines

It is controversial, because obviously root owned tasks are not ptrace-able by the user, and so these tasks will be denied access to the user mounted filesystem (-EACCESS is returned on stat() or any other file operation).

However nobody raised _any_ concrete technical problem associated with this, and the 4 years of widespread use didn't turn up any either. So IMO it's "ugly" only in people's heads and not in reality.

It's ugly ;)

But the problem you're addressing here largely revolves around the fact that the filesystem implementation is a userspace process which is potentially owned by a different user. So you need to prevent the mount owner from peeking at the fs user's activity. That problem is unique to FUSE and so a solution within fuse is appropriate.

This security feature doesn't sounds terribly important to me. So the fuse server can find out what files I'm looking at. But I've already deliberately given the fuse server the ability to ptrace my process?

Can we enhance private namespaces so they can squash setuid/setgid? If so, is that adequate?

#13 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 5 lines

But the problem you're addressing here largely revolves around the fact that the filesystem implementation is a userspace process which is potentially owned by a different user. So you need to prevent the mount owner from peeking at the fs user's activity. That problem is unique to FUSE and so a solution within fuse is appropriate.

It's in fact not so unique to FUSE. It would equally well apply to v9fs or even samba, since both want to allow unprvileged mounts, and synthetic (or at least user-controlled) file serving.

Quoted 3 lines

This security feature doesn't sounds terribly important to me. So the fuse server can find out what files I'm looking at. But I've already deliberately given the fuse server the ability to ptrace my process?

If it's deliberate, than OK.

However with suid/sgid, this is not a deliberate action of the user under whose capabilities the process runs. Neither in the case, when it's a daemon doing some recursive directory traversal.

And it's not just peeking at the filesystem access patterns. A much more dangerous aspect is controlling _when_ an operation returns (e.g. delaying it forever), and _what_ it returns (e.g. huge files/directories).

Of course, this is only truly relevant for systems with untrusted users. But I do want to make FUSE work securely in those cases too.

For the single user system, the sysadmin can turn this feature off, and be done with it.

Quoted 2 lines

Can we enhance private namespaces so they can squash setuid/setgid? If so, is that adequate?

We could. But that would again be overly restrictive. The goal is to make the use of FUSE filesystems for users as simple as possible. If the user has to manage multiple namespaces, each with it's own restrictions, it's becoming a very un-user-friendly environment.

Thanks,
Miklos

#14 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 8 lines · Miklos Szeredi
Quoted 3 lines

Can we enhance private namespaces so they can squash setuid/setgid? If so, is that adequate?

We could. But that would again be overly restrictive. The goal is to make the use of FUSE filesystems for users as simple as possible. If the user has to manage multiple namespaces, each with it's own restrictions, it's becoming a very un-user-friendly environment.

I'd have thought that it would be possible to offer the same user interface as you currently have with private namespaces. Hide any complexity in the userspace tools? Where's the problem?

#15 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 8 lines
Quoted 4 lines

We could. But that would again be overly restrictive. The goal is to make the use of FUSE filesystems for users as simple as possible. If the user has to manage multiple namespaces, each with it's own restrictions, it's becoming a very un-user-friendly environment.

I'd have thought that it would be possible to offer the same user interface as you currently have with private namespaces. Hide any complexity in the userspace tools? Where's the problem?

Sorry, I don't get it.

You mean implement the permission checking in the userspace filesystem? That doesn't work, since it's running with the privileges of the filesystem owner, and as such obviously cannot be trusted with checking for access by other users.

Thanks,
Miklos

#16 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 11 lines · Miklos Szeredi
Quoted 9 lines
Quoted 5 lines

We could. But that would again be overly restrictive. The goal is to make the use of FUSE filesystems for users as simple as possible. If the user has to manage multiple namespaces, each with it's own restrictions, it's becoming a very un-user-friendly environment.

I'd have thought that it would be possible to offer the same user interface as you currently have with private namespaces. Hide any complexity in the userspace tools? Where's the problem?

Sorry, I don't get it.

I'm asking you to expand on what the problems would be if we were to enhance the namespace code as suggested. What's the "very un-user-friendly environment", and why cannot it be made more friendly with appropriate support tools?

#17 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines

We could. But that would again be overly restrictive. The goal is to make the use of FUSE filesystems for users as simple as possible. If the user has to manage multiple namespaces, each with it's own restrictions, it's becoming a very un-user-friendly environment.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I'd have thought that it would be possible to offer the same user interface as you currently have with private namespaces. Hide any complexity in the userspace tools? Where's the problem?

Sorry, I don't get it.

I'm asking you to expand on what the problems would be if we were to enhance the namespace code as suggested.

OK, what I was thinking, is that the user could create a new namespace, that has all the filesystems remounted 'nosuid'. This wouldn't need any new kernel infrastructure, just a suid-root program (e.g. newns_nosuid), that would do a clone(CLONE_NEWNS), then recursively remount everything 'nosuid' in the new namespace. Then restore the user's privileges, and exec a shell.

In this namespace the user could mount things to his heart's content. He could mount over system directories or even the root directory, without being able to do any harm.

This is very nice, but a bit inpractical, since now all the other programs of the user, his desktop environment, login shells etc. Won't be able to see the userspace filesystems mounted in the private namespace.

Of course he can enter the private namespace immediately after login, but then he won't be able to send mail for example, because 'sendmail' could be suid-mail or whatever.

I think the following must be true, for a namespace environment to be useful:

1) All the processes should be able to access the same files (including new mounts)

2) User should be able to run suid/sgid programs

If the suid/sgid programs want to access the contents of a userspace filesystem, that's tough. You can't have that, because it's insecure. But experience shows that this isn't a problem. The exception is the 'mount' program itself, and that is why I'm slowly working towards making sys_mount() and sys_umount() unprivileged.

Miklos

#18 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 22 lines · Miklos Szeredi
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines

We could. But that would again be overly restrictive. The goal is to make the use of FUSE filesystems for users as simple as possible. If the user has to manage multiple namespaces, each with it's own restrictions, it's becoming a very un-user-friendly environment.

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines

I'd have thought that it would be possible to offer the same user interface as you currently have with private namespaces. Hide any complexity in the userspace tools? Where's the problem?

Sorry, I don't get it.

I'm asking you to expand on what the problems would be if we were to enhance the namespace code as suggested.

OK, what I was thinking, is that the user could create a new namespace, that has all the filesystems remounted 'nosuid'. This wouldn't need any new kernel infrastructure, just a suid-root program (e.g. newns_nosuid), that would do a clone(CLONE_NEWNS), then recursively remount everything 'nosuid' in the new namespace. Then restore the user's privileges, and exec a shell.

Quoted 8 lines

In this namespace the user could mount things to his heart's content. He could mount over system directories or even the root directory, without being able to do any harm.

This is very nice, but a bit inpractical, since now all the other programs of the user, his desktop environment, login shells etc. Won't be able to see the userspace filesystems mounted in the private namespace.

Yup, that's useless. That makes the whole CLONE_NEWNS idea unworkable, yes?

Have we exhausted the alternatives?

(If, as you say, v9fs and samba (did you mean smbfs/cifs?) want unprivileged mounts, shouldn't the code which you have there be moved out of fs/fuse/ and into fs/?)

#19 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 2 lines

Yup, that's useless. That makes the whole CLONE_NEWNS idea unworkable, yes?

For solving this problem, yes.

Quoted 1 line

Have we exhausted the alternatives?

I believe yes. At least to date no workable alternative has been suggested.

Quoted 3 lines

(If, as you say, v9fs and samba (did you mean smbfs/cifs?) want unprivileged mounts, shouldn't the code which you have there be moved out of fs/fuse/ and into fs/?)

Probably. But since the code is so small, and there's no other user yet, I didn't want to take the initiative.

Thanks,
Miklos

#20 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

In reply to Miklos Szeredi
Quoted 11 lines
Quoted 3 lines

I'm asking you to expand on what the problems would be if we were to enhance the namespace code as suggested.

OK, what I was thinking, is that the user could create a new namespace, that has all the filesystems remounted 'nosuid'. This wouldn't need any new kernel infrastructure, just a suid-root program (e.g. newns_nosuid), that would do a clone(CLONE_NEWNS), then recursively remount everything 'nosuid' in the new namespace. Then restore the user's privileges, and exec a shell.

I'm confused why everything has to be remounted nosuid. I understand enforcing synthetics to be mounted nosuid, but not the rest of the file systems. I thought all the problems revolving around the private namespace solution where the FUSE team's desire to have per-user namespace and/or per-session namespace versus per-process namespace.

-eric

#21 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 11 lines
Quoted 10 lines
Quoted 2 lines

I'm asking you to expand on what the problems would be if we were to enhance the namespace code as suggested.

OK, what I was thinking, is that the user could create a new namespace, that has all the filesystems remounted 'nosuid'. This wouldn't need any new kernel infrastructure, just a suid-root program (e.g. newns_nosuid), that would do a clone(CLONE_NEWNS), then recursively remount everything 'nosuid' in the new namespace. Then restore the user's privileges, and exec a shell.

Quoted 3 lines

I'm confused why everything has to be remounted nosuid. I understand enforcing synthetics to be mounted nosuid, but not the rest of the file systems.

It's related to the problem of a suid program accessing synthetic filesystem, and filesystem doing something bad to suid program (make it hang, supply bogus data ...). This can be solved by "squashing" suid for the whole namespace (basically the Plan 9 solution). Unfortunately this is not really practical in Linux/Unix.

Quoted 3 lines

I thought all the problems revolving around the private namespace solution where the FUSE team's desire to have per-user namespace and/or per-session namespace versus per-process namespace.

That's a different aspect of this thing. Private namespaces cannot do anything about the above problem.

Miklos

#22 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 11 lines · Miklos Szeredi
Quoted 4 lines

I'm confused why everything has to be remounted nosuid. I understand enforcing synthetics to be mounted nosuid, but not the rest of the file systems.

It's related to the problem of a suid program accessing synthetic filesystem, and filesystem doing something bad to suid program (make it hang, supply bogus data ...). This can be solved by "squashing" suid for the whole namespace (basically the Plan 9 solution). Unfortunately this is not really practical in Linux/Unix.

Just to make sure I understand you - if I don't squash suid for the entire name space, a user could mount a malicious synthetic (even with NOSUID) and then launch an SUID app from an inherited mount which would then traverse to the malicious synthetic.

That's a nasty case I hadn't considered before -- however, what's the potential damage there? The user could hold up progress of the SUID app that they launched, but that wouldn't necessarily impede system progress since system-critical suid apps wouldn't be typically launched by a user. I suppose there is the possibility that if multiple instances of such an SUID app share a global lock you could get into trouble -- do we have any concrete example apps that would exhibit this kind of behavior?

Are there other vunerabilities that I'm missing?

-eric

#23 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 11 lines
Quoted 6 lines

It's related to the problem of a suid program accessing synthetic filesystem, and filesystem doing something bad to suid program (make it hang, supply bogus data ...). This can be solved by "squashing" suid for the whole namespace (basically the Plan 9 solution). Unfortunately this is not really practical in Linux/Unix.

Just to make sure I understand you - if I don't squash suid for the entire name space, a user could mount a malicious synthetic (even with NOSUID) and then launch an SUID app from an inherited mount which would then traverse to the malicious synthetic.

Yes.

Quoted 8 lines

That's a nasty case I hadn't considered before -- however, what's the potential damage there? The user could hold up progress of the SUID app that they launched, but that wouldn't necessarily impede system progress since system-critical suid apps wouldn't be typically launched by a user. I suppose there is the possibility that if multiple instances of such an SUID app share a global lock you could get into trouble -- do we have any concrete example apps that would exhibit this kind of behavior?

I don't know any. But with 'sudo' the potential set of SUID apps is basically infinite, so you'd have a hard time proving that this sort of situation won't arise.

Quoted 1 line

Are there other vunerabilities that I'm missing?

Another theoretical possibility is that you make the SUID app consume some resource by feeding it a large-file/deep-directory/etc that quota would otherwise prevent (you can't do quota on a synthetic filesystem, without the filesystem's cooperation).

Miklos

#24 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

In reply to Miklos Szeredi

Hi!

Quoted 7 lines
Quoted 2 lines

Can we enhance private namespaces so they can squash setuid/setgid? If so, is that adequate?

We could. But that would again be overly restrictive. The goal is to make the use of FUSE filesystems for users as simple as possible. If the user has to manage multiple namespaces, each with it's own restrictions, it's becoming a very un-user-friendly environment.

Actually I think this solution is way less ugly. We have precent: if task is ptraced, suid bits on anything it execs are ignored.

I don't think user interface issues belong to the kernel (and I do not think different namespaces are that bad for the user; various chroots and ld_preload hacks already work that way)
Pavel

Signature · 1 line
teflon -- maybe it is a trademark, but it should not be.
#25 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

In reply to Miklos Szeredi
Quoted 7 lines · Miklos Szeredi

The controversial part is fuse_allow_task() called from fuse_permission() and fuse_revalidate() (fs/fuse/dir.c).

This function (as explained by the header comment) disallows access to the filesystem for any task, which the filesystem owner (the user who did the mount) is not allowed to ptrace.

Quoted 8 lines

The rationale is that accessing the filesystem gives the filesystem implementor ptrace like capabilities (detailed in Documentation/filesystems/fuse.txt)

It is controversial, because obviously root owned tasks are not ptrace-able by the user, and so these tasks will be denied access to the user mounted filesystem (-EACCESS is returned on stat() or any other file operation).

I don't think this should be objectionable, since we already have times when root-owned tasks can get EACCESS when accessing some filesystem. This can happen with any distributed filesystem that enforces real security --- whether it be nfs-root-squash, or the Andrew Filesystem, or NFSv4. Root can get "permission denied" when some other userid with appropriate credentials would be allowed to access the file/directory.

On the other hand, sometimes a root process, or some other user's process, might _want_ to give permission to allow a trusted FUSE filesystem the potential to monkey with it (return potentially untrusted information, or stop it entirely), in exchange for access to the filesystem. So it would be nice if there was some way that a process could tell the kernel that it is willing to give permission to allow certain FUSE filesystems to potentially affect it. Say, via a fnctl() call, perhaps.

- Ted

#26 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 7 lines

I don't think this should be objectionable, since we already have times when root-owned tasks can get EACCESS when accessing some filesystem. This can happen with any distributed filesystem that enforces real security --- whether it be nfs-root-squash, or the Andrew Filesystem, or NFSv4. Root can get "permission denied" when some other userid with appropriate credentials would be allowed to access the file/directory.

Right.

Quoted 8 lines

On the other hand, sometimes a root process, or some other user's process, might _want_ to give permission to allow a trusted FUSE filesystem the potential to monkey with it (return potentially untrusted information, or stop it entirely), in exchange for access to the filesystem. So it would be nice if there was some way that a process could tell the kernel that it is willing to give permission to allow certain FUSE filesystems to potentially affect it. Say, via a fnctl() call, perhaps.

Hmm. 'su' works for root.

How do you think fcntl() could be used? I think a task flag settable via prctl() would be more appropriate.

Miklos

#27 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 10 lines · Miklos Szeredi
Quoted 8 lines

On the other hand, sometimes a root process, or some other user's process, might _want_ to give permission to allow a trusted FUSE filesystem the potential to monkey with it (return potentially untrusted information, or stop it entirely), in exchange for access to the filesystem. So it would be nice if there was some way that a process could tell the kernel that it is willing to give permission to allow certain FUSE filesystems to potentially affect it. Say, via a fnctl() call, perhaps.

Hmm. 'su' works for root.

?? Not sure what you're saying.

Quoted 2 lines

How do you think fcntl() could be used? I think a task flag settable via prctl() would be more appropriate.

The problem with a task flag is that a root process might not want to give permission for _all_ user-mountable filesystem, but only certain ones. So the idea is that the process should be able to specify specific mountpoints as being "ok" for the process to access.

- Ted

#28 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 9 lines
Quoted 9 lines
Quoted 8 lines

On the other hand, sometimes a root process, or some other user's process, might _want_ to give permission to allow a trusted FUSE filesystem the potential to monkey with it (return potentially untrusted information, or stop it entirely), in exchange for access to the filesystem. So it would be nice if there was some way that a process could tell the kernel that it is willing to give permission to allow certain FUSE filesystems to potentially affect it. Say, via a fnctl() call, perhaps.

Quoted 3 lines
Quoted 1 line

Hmm. 'su' works for root.

?? Not sure what you're saying.

If user X mounts a filesystem, and root wants to access it, it can do 'su X' and see the otherwise inaccesible files.

Quoted 7 lines
Quoted 2 lines

How do you think fcntl() could be used? I think a task flag settable via prctl() would be more appropriate.

The problem with a task flag is that a root process might not want to give permission for _all_ user-mountable filesystem, but only certain ones. So the idea is that the process should be able to specify specific mountpoints as being "ok" for the process to access.

OK, so you're saying, it should be a per-mountpoint flag, and not a per-process one.

Then the solution is simple I think. Root can just remount the filesystem with the 'allow_other' flag (that's currently not possible, because the remount_fs operation is not yet implemented).

Thanks,
Miklos

#29 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 4 lines · Miklos Szeredi
Quoted 1 line

?? Not sure what you're saying.

If user X mounts a filesystem, and root wants to access it, it can do 'su X' and see the otherwise inaccesible files.

That's rather awkward, and will be painful in certain applications that are trying to scan multiple filesystems, possibly run by different users X, Y, and Z.

Quoted 10 lines
Quoted 7 lines
Quoted 2 lines

How do you think fcntl() could be used? I think a task flag settable via prctl() would be more appropriate.

The problem with a task flag is that a root process might not want to give permission for _all_ user-mountable filesystem, but only certain ones. So the idea is that the process should be able to specify specific mountpoints as being "ok" for the process to access.

OK, so you're saying, it should be a per-mountpoint flag, and not a per-process one.

No, I'm saying that it should be both per-mountpoint and per-process. Hence something like fnctl. Problem is though fnctl() works on a file descriptor, and with FUSE running you wouldn't be able to even get a file descriptor opened on the mountpoint.

Quoted 3 lines

Then the solution is simple I think. Root can just remount the filesystem with the 'allow_other' flag (that's currently not possible, because the remount_fs operation is not yet implemented).

Globally remounting the filesystem allow_other isn't the right thing, since applications that don't know what to expect will be taken by surprise. Hence my suggestion to do something which is both per-mountpoint and per-process.

- Ted

#30 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

In reply to Andrew Morton

EHLO,

Quoted 8 lines · Andrew Morton
Quoted 4 lines
Quoted 2 lines

System where users can mount their own filesystems should not be called "Unix" any more.

It's not. It's "Linux".

It would be helpful if we could have a brief description of the feature which you're discussing here. We discussed this a couple of months back, but I've forgotten most of it and it was off-list I think.

<offtopic>

Excuse me, I'm far from being a filesystem/vfs expert ... However I've got the idea about the merging fuse/reiser4 that some guys keep complaining about the quite strange behaviour of these stuffs: when I write 'strange' I mean strange from the view point of some standard unix ideas about filesystems
(and anything related to filesystems like permission checking, namespaces
etc) and how they should be implemented and handled.

This reminds me articles about comparing Linux and BSDs. BSD guys claims that BSD distros _ARE_ unices but Linux is not. It's out of scope to waste mails about these flames like this (it's question of view point as almost always) however there IS some lesson here. BSD systems are somewhat (well, not counting the interesting ideas of DragonFly BSD) conservative to implement new stuffs. I'm about stuffs like filesystem transactions, API exported to the user space to be able to do things like deleting data from the begining of the file (there is API call to truncate - from the end of the file ...) and such (what a quite braindead idea to rewrite eg a 10Gbyte long file just for inserting one byte to somewhere in the middle of the file
- in 2005 ...). The only thing explains where the later is not present in most OSes is because of historical reasons and not technical ones. And if even Linux does not want to open toward extended filesystem abilities which common open source system will? I guess none.

I can inmagine that vendors of some closed source systems will exploit the hole in the area of outdated filesystem concept of our current world and when it becomes reality it's to late. Maybe.

Please forgive for my possible offtopic mail here but I could not resist :)

</offtopic>

Signature · 1 line
- Gábor
#31 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

In reply to Miklos Szeredi

Hi!

Quoted 8 lines
Quoted 7 lines
Quoted 5 lines

You have a choice of: 1) believe me that the current solution is fine

2) get down and try to understand the damn thing, and then come up with technical arguments for/against it

Argument is "it is **** ugly".

Quoted 7 lines

Yeah, that's your opinion. Mine is that it's f****** beautiful ;).

There are plenty of ugly things in Unix/Linux that you've become so accustomed to, that they no longer seem ugly. Think about the sticky bit on directories for example. That one was breaking assumptions left and right when it got introduced, but people came to accept it, because it's useful.

Just for the record, I still consider sticky bit "slightly" ugly and nfs root squash "very" ugly.

Quoted 7 lines
Quoted 3 lines

Your fuse.txt explains why it is not security hole. It does not explain why your interface is the best possible, and what alternative ways of "not security hole" exist.

That's because I don't see any alternative. The "preventing user from tracing root" and "preventing access to user's filesysem by root" must come together. There's doesn't seem to be any other way.

It is clear that we can't allow root (or anyone else) to access that filesystem. Infinite namespace is nice trap.

Quoted 1 line

BTW, thanks for reading through fuse.txt :)

You are welcome ;-).
Pavel

Signature · 1 line
teflon -- maybe it is a trademark, but it should not be.
#32 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

In reply to Pavel Machek
Quoted 2 lines · Pavel Machek

How is it going
to interact with backup tools?

I admit to severe dislike for any backup tool which crosses file system borders without permission, and therefore am not going to cry if something breaks because of this.

Ditto updatedb/locate -- it's a remote file system. Tools should treat it as such.

NB: I am assuming that root can stat() the mountpoint and that it's not possible to delay this call arbitrarily. If not, the debate IMHO should be whether to special-case this situation.

Signature · 4 lines
Matthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  [email protected]
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
 - -
The early worm has a death wish.
#33 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

In reply to Pavel Machek
Quoted 8 lines · Pavel Machek
Quoted 2 lines

An emotional argument again. What's "strange" about it?

Not so emotional argument...

System where users can mount their own filesystems should not be called "Unix" any more. It introduces new mechanism, similar to ptrace.

I think that's a rather severe statement. I don't see allowing user
mounts damaging standard UNIX system semantics as long as certain
rules are followed. After all, user-mounts and private name spaces
are what the original authors of UNIX went on to develop.

Quoted 4 lines

It restricts root in ways not seen before. How is updatedb/locate supposed to work on system with this? How is it going to interact with backup tools?

I think requiring user-mounts to be in private name spaces solves many of these issues -- you don't have to worry about system daemons and/or other users wandering into synthetic file systems.

-eric

#34 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Hi!

Quoted 4 lines
Quoted 4 lines
Quoted 1 line

An emotional argument again. What's "strange" about it?

Not so emotional argument...

Quoted 8 lines
Quoted 3 lines

System where users can mount their own filesystems should not be called "Unix" any more. It introduces new mechanism, similar to ptrace.

I think that's a rather severe statement. I don't see allowing user
mounts damaging standard UNIX system semantics as long as certain
rules are followed. After all, user-mounts and private name spaces
are what the original authors of UNIX went on to develop.

Well, but notice how it is called "Plan 9", not "Unix" :-). The "certain rules" are rather tricky to enforce... btw any ideas how Plan 9 solves problems around user-mounts? Does it allow it at all?

Pavel

Signature · 1 line
teflon -- maybe it is a trademark, but it should not be.
#35 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 10 lines · Pavel Machek
Quoted 5 lines

I think that's a rather severe statement. I don't see allowing user
mounts damaging standard UNIX system semantics as long as certain
rules are followed. After all, user-mounts and private name spaces
are what the original authors of UNIX went on to develop.

Well, but notice how it is called "Plan 9", not "Unix" :-). The "certain rules" are rather tricky to enforce... btw any ideas how Plan 9 solves problems around user-mounts? Does it allow it at all?

I agree the rules can be tricky, but I think we can start with something ultra-conservative (but not as ultra-conservative as no user mounts) and work from there.

Plan 9 does indeed allow (in fact in many ways it depends on) user mounts. There are a number of answers for how it gets around some of the sticky issues -- some of those may be applicable in UNIX-style systems, while others most likely don't apply.

The ones that don't apply:
(a) Plan 9 has no root user, nor does it have a concept of setuid/setgid -- this solves a lot of problems, but isn't practical in UNIX environments.
(b) Plan 9 cluster environments typically have user-controller terminals and shared centralized resources (cpu servers, file servers, etc.) Users have more or less complete control over their local workstation (which includes their name space) -- and they can import resources/services from shared servers (where they have less/no control). Critical services, such as authentication, are hosted on a remote server protected from normal users. While this sort of environment can be set up using UNIX systems -- its probably not practical for most folks.

The ones that do apply:
(c) Private Name Space - by default, every new process in Plan 9 gets a private name space inherited from its parent. This means that any user mounts will be in their own private name space and not interfere with system tasks or other users. As I've stated, I think this has direct application in UNIX environments. We may not go to the extreme of making CLONENS the default for new threads, but I think it should be easy enough to enforce unprivileged mounts be in a non-global name space.
(d) For situations in which sharing a name space is desirable, the name space is re-exported through a global handle (/srv in Plan 9's case) so that a user can mount instances of his private name space in other threads -- or he can set permissions on the handle so it may be used by other users. This re-export is done using the 9P protocol (although if re-exporting to the same system it short-circuits to just function calls within the OS). The 9P layer adds a certain level of security to just allowing name spaces to be accessed/mounted directly
- of course it also adds a bit of overhead. We are currently looking at this sort of implementation in the v9fs 2.1 release.

Given that (a) & (b) aren't practical in UNIX environments, I think a reasonable set of restrictions (some of which were suggested by Miklos) can be used to maintain integrity:
1) only allow user's to mount/bind on directories/files where they have unconditional write access.
2) enforce NOSUID mount options on user-mounts

If you combine these two restrictions with only allowing unprivileged mounts in private name space I think you get 90% there. The only thing left to resolve is the best way to allow sharing private name spaces between threads/users -- and I still view this as more of extended functionality than a hard-requirement.

I'm giving a talk at OLS overviewing my perspective on the barriers and solutions. My motivation was to try and get some dialog/compromise/closure on some of the user-space mount and private name space issues. Hopefully some of the stake-holders will attend.

-eric

#36 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

On 6/22/05, Eric Van Hensbergen <[email protected]> wrote:

    ...
Quoted 7 lines

If you combine these two restrictions with only allowing unprivileged mounts in private name space I think you get 90% there. The only thing left to resolve is the best way to allow sharing private name spaces between threads/users -- and I still view this as more of extended functionality than a hard-requirement.

Reviewing my notes, there were a few subtle restrictions I forgot (most of which originally suggested by Miklos):

(a) User's can't mount file system types not deemed "safe" (via flag in the file system type) -- this should help mitigate user's exploiting bugs in existing file systems to interfere with the system. Judging when a file system type is "safe" is a nasty kettle of fish
though...
(b) Enforce NODEV along with NOSUID so that user-based synthetics can't have device inodes with compromised permissions, etc.

-eric

#37 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

In reply to Eric Van Hensbergen
Quoted 2 lines

1) only allow user's to mount/bind on directories/files where they have unconditional write access.

Like say /tmp. Build a bizarre behaving /tmp and I can do funky stuff with some third party suid apps. Its a good start but you probably want a stronger policy and one enforced by the user space side not kernel (eg "Below ~")

Quoted 1 line

2) enforce NOSUID mount options on user-mounts

2 is unneccessarily crude. Just enforce suid owner/owner group.

Alan

#38 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 8 lines · Alan Cox
Quoted 2 lines

1) only allow user's to mount/bind on directories/files where they have unconditional write access.

Like say /tmp. Build a bizarre behaving /tmp and I can do funky stuff with some third party suid apps. Its a good start but you probably want a stronger policy and one enforced by the user space side not kernel (eg "Below ~")

Well in the original discussions Miklos had classified directories that had the sticky bit set (such as /tmp) as out-of-bounds for
user-mounts. However, its a point well taken. I had originally
proposed having some sort of a policy file (sort of like an extended fstab with regular expressions) to give more granular control over where users could and couldn't mount (along with what types of devices, network servers, and file systems they could mount from). However, this leans more towards the "super-mount" suid-application which I think many found undesirable. An alternative would be some way for the kernel to consult with an application about different mount policies. I don't know what the right thing is here.

Quoted 4 lines
Quoted 1 line

2) enforce NOSUID mount options on user-mounts

2 is unneccessarily crude. Just enforce suid owner/owner group.

I'm dense this morning, not sure what you mean here.

-eric

#39 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

In reply to Miklos Szeredi
Quoted 5 lines · Miklos Szeredi

I won't shed many tears if you drop fuse-nfs-export.patch. It would at least give the userspace solution some boost.

However the patch is pretty small, and despite it's flaws, I know it's used by a number of people.

Why not leave it up to the user as an option, for the time being at least. Does this somehow break things?

thanks,
avuton

Signature · 1 line
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
#40 lore raw

Re: -mm -> 2.6.13 merge status (fuse)

Quoted 8 lines
Quoted 5 lines

I won't shed many tears if you drop fuse-nfs-export.patch. It would at least give the userspace solution some boost.

However the patch is pretty small, and despite it's flaws, I know it's used by a number of people.

Why not leave it up to the user as an option, for the time being at least.

Making it a config option could make sense, yes.

Quoted 1 line

Does this somehow break things?

You mean outside NFS export? No, it's completely harmless.

NFS export itself is slightly broken (random ESTALE errors), but it's still useful.

Thanks,
Miklos

#41 lore raw
In reply to Andrew Morton
Quoted 38 lines · Andrew Morton

This summarises my current thinking on various patches which are presently in -mm. I cover large things and small-but-controversial things. Anything which isn't covered here (and that's a lot of material) is probably a "will merge", unless it obviously isn't.

(If you reply to this email it would be a good idea to alter the Subject:
to reflect which feature you are discussing)

git-ocfs

The OCFS2 filesystem. OK by me, although I'm not sure it's had enough review.

sparsemem

OK by me for a merge. Need to poke arch maintainers first, check that they've looked at it sufficiently closely.

vm-early-zone-reclaim

Needs some convincing benchmark numbers to back it up. Otherwise OK.

avoiding-mmap-fragmentation

Tricky. Addresses vm area fragmentation issues due to recent optimisations to the free-area lookup code. Will merge.

periodically-drain-non-local-pagesets

Will merge

pcibus_to_node and users

Will merge

Quoted 8 lines

CONFIG_HZ for x86 and ia64: changes default HZ to 250, make HZ Kconfigurable.

Will merge (will switch default to 1000 Hz later if that seems necessary)

dmi-*.patch

Will merge. I have a comment "The below break x440". Maybe it got fixed. We'll doubtless hear if not.

Fixed, patch merged in -mm as dmi-move-acpi-sleep-quirk-fix.patch

http://marc.theaimsgroup.com/?l=linux-kernel&m=111829134708641&w=2
http://marc.theaimsgroup.com/?l=linux-kernel&m=111832375203467&w=2

Quoted 160 lines

xen-*.patch

These are little cleanups and abstractions which make a Xen merge easier. May as well merge them.

CPU hotplug for x86 and x86_64

Not really useful on current hardware, but these provide infrastructure which some power management patches need, and it seems sensible to make the reference architecture support hotplug. Will merge.

swsusp-on-SMP

Will merge.

cfq version 3

Not sure. Jens seems to be setting up a few git trees. On hold.

RCUification of the key management code

Don't know - dhowells seemed diffident last time we discussed this.

timers-fixes-improvements.patch

SMP speedups for the core timer code. It was bumpy, but this seems stable now. Will merge.

kprobes-*

Will merge

rapidio-*

Will merge.

namespace*.patch

Awaiting viro ack.

xtensa architecture

Is xtensa now, or will it be in the future a sufficiently popular architecture to justify the cost of having this code in the tree?

Heaven knows. Will merge.

dlm-*.patch: Red Hat distributed lock manager

Hard. Right now it seems that no in-kernel projects will use this and only one out-of-kernel project will use it. Shelve the problem until after Kernel Summit, where some light may be shed.

Opinions are sought...

connector.patch

Nice idea IMO, but there are still questions around the implementation. More dialogue needed ;)

connector-add-a-fork-connector.patch

OK, but needs connector.

inotify

There are still concerns about the userspace API and internal implementation details. More slogging needed.

pcmcia-*.patch

Makes the pcmcia layer generate hotplug events and deprecates cardmgr. Will merge.

NUMA-aware slab allocator

Seems stable now, but it needs some ifdef reduction work before merging, please.

CPU scheduler

Will merge some of these patches. We're still discussing which ones.

perfctr

Not yet, but getting closer. The PPC64 guys still need to sort out a few interface issues with Mikael. We might be able to fit this into 2.6.13 if people get a move on.

cachefs

This is a ton of code which knows rather a lot about pagecache internals. It allows the AFS client to cache file contents on a local blockdev.

I don't think it's a justified addition for only AFS and I'd prefer to see it proven for NFS as well.

Issues around add-page-becoming-writable-notification.patch need to be resolved.

cachefs-for-nfs

A recent addition. Needs review from NFS developers and considerably more testing.

These things aren't looking likely for 2.6.13.

kexec and kdump

I guess we should merge these.

I'm still concerned that the various device shutdown problems will mean that the success rate for crashing kernels is not high enough for kdump to be considered a success. In which case in six months time we'll hear rumours about vendors shipping wholly different crashdump implementations, which would be quite bad.

But I think this has gone as far as it can go in -mm, so it's a bit of a punt.

reiser4

Merge it, I guess.

The patches still contain all the reiser4-specific namespace enhancements, only it is disabled, so it is effectively dead code. Maybe we should ask that it actually be removed?

v9fs

I'm not sure that this has a sufficiently high usefulness-to-maintenance-cost ratio.

fuse

This is useful, but there are, AFAIK, two issues:

- We're still deadlocked over some permission-checking hacks in there

- It has an NFS server implementation which only works if the to-be-served file happens to be in dcache.

It has been said that a userspace NFS server can be used to get full NFS server functionality with FUSE. I think the half-assed kernel implementation should be done away with.

execute-in-place

Will merge. Have the embedded guys commented on the usefulness of this for execute-out-of-ROM?

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 2 lines
Andrey Panin		| Linux and UNIX system administrator
[email protected]		| PGP key: wwwkeys.pgp.net
#42 lore raw
In reply to Andrew Morton
Quoted 2 lines · Andrew Morton

CONFIG_HZ for x86 and ia64: changes default HZ to 250, make HZ Kconfigurable.
Will merge (will switch default to 1000 Hz later if that seems necessary)

This has been in Fedora for a while. DaveJ can probably give you more info. From own testing 100Hz is how far down you want to go to avoid random clock slew on laptops and to see power improvements.

#43 lore raw
Quoted 7 lines · Alan Cox
Quoted 2 lines · Andrew Morton

CONFIG_HZ for x86 and ia64: changes default HZ to 250, make HZ Kconfigurable.
Will merge (will switch default to 1000 Hz later if that seems necessary)

This has been in Fedora for a while. DaveJ can probably give you more info. From own testing 100Hz is how far down you want to go to avoid random clock slew on laptops and to see power improvements.

actually 250Hz is a not so fun value. 300 is a lot nicer (multiple of both 50Hz and 60Hz and thus covers most TV standards)

#44 lore raw
Quoted 11 lines · Arjan van de Ven
Quoted 7 lines · Alan Cox
Quoted 2 lines · Andrew Morton

CONFIG_HZ for x86 and ia64: changes default HZ to 250, make HZ Kconfigurable.
Will merge (will switch default to 1000 Hz later if that seems necessary)

This has been in Fedora for a while. DaveJ can probably give you more info. From own testing 100Hz is how far down you want to go to avoid random clock slew on laptops and to see power improvements.

actually 250Hz is a not so fun value. 300 is a lot nicer (multiple of both 50Hz and 60Hz and thus covers most TV standards)

Sorry, the ITU-R "M" standard specifies 30000/1001 frames (60000/1001 fields) per second, that's close to 30/60 but not the same. Now please please don't make use run HZ = (60000/1001 * 5) or similar ;-)

BTW it seems to me that power management using C2/C3 states will work much more efficiently with a lower HZ because the chipset/processor will be a larger percentage of the time in c2/c3 mode, right?

#45 lore raw
In reply to Andrew Morton
Quoted 4 lines · Andrew Morton

execute-in-place

Will merge. Have the embedded guys commented on the usefulness of this for execute-out-of-ROM?

Allright. Going to push our test-team to run their tests on the xip code that is in -mm.

#46 lore raw
In reply to Andrew Morton
Quoted 5 lines · Andrew Morton

execute-in-place

Will merge. Have the embedded guys commented on the usefulness of this for execute-out-of-ROM?

It looks fairly useful, but XIP for NOR flashes still needs additional work. No objections from my side.

Jörn

Signature · 3 lines
Optimizations always bust things, because all optimizations are, in
the long haul, a form of cheating, and cheaters eventually get caught.
-- Larry Wall 
#47 lore raw

Re: -mm -> 2.6.13 merge status (Suspend-to-disk)

In reply to Andrew Morton

Hi.

(Marcelo: Copied for issue at the bottom).

Quoted 5 lines · Andrew Morton

CPU hotplug for x86 and x86_64

Not really useful on current hardware, but these provide infrastructure which some power management patches need, and it seems sensible to make the reference architecture support hotplug. Will merge.

Yay. I'm not going to use it yet, but know Pavel wants it for the next one.

Quoted 8 lines

swsusp-on-SMP

Will merge.

kexec and kdump

I guess we should merge these.

Quoted 8 lines

I'm still concerned that the various device shutdown problems will mean that the success rate for crashing kernels is not high enough for kdump to be considered a success. In which case in six months time we'll hear rumours about vendors shipping wholly different crashdump implementations, which would be quite bad.

But I think this has gone as far as it can go in -mm, so it's a bit of a punt.

No potential clashes with suspend code, I assume?

Quoted 4 lines

execute-in-place

Will merge. Have the embedded guys commented on the usefulness of this for execute-out-of-ROM?

Switch roles for a mo and put my Cyclades hat on. Probably not useful to us at the moment, at least in the case of the products I work on. Marcelo?

Regards,

Nigel

#48 lore raw

Re: -mm -> 2.6.13 merge status (Suspend-to-disk)

Hi!

Quoted 4 lines
Quoted 4 lines

kexec and kdump

I guess we should merge these.

...

Quoted 2 lines

No potential clashes with suspend code, I assume?

I test suspend in -mm series from time to time, and it seems ok; so this one should be safe.
Pavel

Signature · 1 line
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         
#49 lore raw

Re: -mm -> 2.6.13 merge status (Suspend-to-disk)

In reply to Nigel Cunningham
Quoted 9 lines
Quoted 5 lines

execute-in-place

Will merge. Have the embedded guys commented on the usefulness of this for execute-out-of-ROM?

Switch roles for a mo and put my Cyclades hat on. Probably not useful to us at the moment, at least in the case of the products I work on. Marcelo?

Well yes, its definately very useful for embedded folks where RAM is a precious resource (not our case at the moment).

I'm not aware of any users of this XIP implementation, maybe Tim Bird or Russell have reviewed/tested it?

It went through filesystem folks reviewing (and I'm pretty sure akpm knows about that already)...

Hope to be helpful.

#50 lore raw

Re: v9fs (-mm -> 2.6.13 merge status)

In reply to Andrew Morton
Quoted 6 lines · Andrew Morton

v9fs

I'm not sure that this has a sufficiently high usefulness-to-maintenance-cost ratio.

I think v9fs/9P has some unique aspects which differentiate it from the other distributed system protocols integrated into Linux:
a) it presents a unified distributed resource sharing protocol. It will be able to distribute devices, file systems, system services, and application interfaces.
b) it provides non-caching RPC-style access to synthetic file systems which could be used with in-kernel file systems such as sysfs or with user-space synthetics such as those provided by FUSE
c) its implementation supports transport independence enabling easy support for different interconnects (shared memory, Xen device channels, RDMA, Infiniband, etc.)

v9fs-2.0 has a somewhat limited audience at the moment - but now that the initial implementation is more or less complete we are working to build applications on top of it (and provide a better server). It's being integrated into cluster projects at LANL and being looked at wrt virtualization I/O at IBM. Its our hope that these improvements and cluster applications will motivate more wide-spread use of the v9fs module.

-eric

#51 lore raw

Re: v9fs (-mm -> 2.6.13 merge status)

Quoted 6 lines · Eric Van Hensbergen
Quoted 5 lines · Andrew Morton

v9fs

I'm not sure that this has a sufficiently high usefulness-to-maintenance-cost ratio.

The 9P protocol implemented by v9fs is the result of over a decade of research in distributed systems at Bell Labs by the original Unix team, and it has various implementations for other operating systems that have been used in production systems for many years.

9P is designed to be portable across systems and transport protocols, it's network transparent, and it gives us interoperativity with Inferno(which can run hosted under Linux already), Plan 9, and p9p, and implementations for *BSD and other systems are in the works.

9P has the potential to become the standard protocol for distributed resources and I don't think any of the alternatives come anywhere near being as well designed, well proven and encompassing.

uriel

#52 lore raw

Re: v9fs (-mm -> 2.6.13 merge status)

Quoted 3 lines · Uriel <uriell <at> binarydream.org> writes:

..
The 9P protocol implemented by v9fs is the result of over a decade of research in distributed systems at Bell Labs by the original Unix team,

I would second that. There are several other filesystems already merged or under discussion that are superficially similar - fuse, coda, and even nfs! However, v9fs is the only one that is (all of)
1) suitable for synthetic filesystems (resources),
as well as 'normal' filesystems
2) equally good for local and remote filesystems
3) OS/machine independent "on-the-wire"
4) (network) transport independent
5) has a significant history of deployment

If there are few Linux applications using v9fs as yet (but there is plan9ports), then one must admit that there is something of a chicken-and-eggs situation involved!

Martin

A message that is not in the archive (sent off-list or not yet received): [email protected]

#54 lore raw

Re: Fwd: v9fs (-mm -> 2.6.13 merge status)

Quoted 7 lines · Eric Van Hensbergen
Quoted 6 lines · Andrew Morton

v9fs

I'm not sure that this has a sufficiently high usefulness-to-maintenance-cost ratio.

I got pointed at this discussion. Here are my $.02 on why we at LANL are interested in v9fs.

We build clusters on the order of 2000 machines at present, with larger systems coming along. The system which we use to run these clusters is bproc. While bproc has proven to be very powerful to date, it does have its limits:
- requires homogenous system
- the network protocols it uses, while simple, are somewhat ad-hoc (as is common in this type of system)
- if you are on a bproc system as user x, using 25% of the system, you still see 100% of the processes. This is a bit of a security issue.

We have a desire to build single-system-image looking clusters along the bproc model, but at the same time compose those clusters of, e.g., Opterons and G5s. This mixing is highly desirable for compoutations that have phases, some of which belong on one type of a machine, and some on another.

We are going to use v9fs as the glue for our next-generation cluster software, called 'xcpu'. Xcpu has been implemented on Plan 9 and works there. I have ported xcpu to Linux, using v9fs as the client side and Russ Cox's plan9ports server to write servers.

xcpu presents a remote execution service as a 9p server. xcpu has been tested across architectures and it works very well. By summer 2006, we hope to have cut over our bproc systems to xcpu.

That's one use for v9fs. We also plan to use v9fs to provide us with servers for global /proc, monitoring, and control systems for our clusters.

The global /proc is interesting. bproc provides a global /proc, but it is incomplete; entries for, e.g., exe and maps are not filled in. bproc also caches part of the /proc, but the rules about what is cached and what the timeouts are, are set in the kernel module and not easily changed. We are going to have an "aggregating" user level 9p server based on Mirtchovskis's aggrfs, which will both aggregate all the cluster nodes, and have caching rules that make sense in clusters of 1000s of node (for example, it is ok to cache /proc/x/status; there is no need to cache /proc/x/maps, and you probably don't want to anyway).

A neat capability is that if we give a user, e.g., 25% of the cluster, we can tailor that user's name space so that they only see their procs and the 25% of the cluster they own. This is good for security, but also good for convenience: most users don't really care that some other user is on 75% of the cluster. Global pid spaces are neat in theory, messy in practice at large scale. I want my global pid space to be global to *me*, meaning I see the global space of the nodes I care about. The sysadmin, of course, wants to see everything. All this is possible. V9fs, along with Linux private name spaces, will allow us to provide this model: users can see some or all of the global pid space, depending on need; users can be constrained to only see part of the global pid space, depending on other issues.

9p will also replace the Supermon protocol, allowing people to easily view status information in a file system.

In addition to the cluster usage, there is also grid usage. The 9grid, composed of plan 9 systems, is connected by 9p servers. Linux systems can join the 9grid with no problem, once Linux has v9fs.

Were v9fs just a file system, I would not really be interested in it one way or another; we have NFS, after all. But v9fs is really the key piece of a new model of cluster services we are building at LANL. 9p will be the glue, and v9fs will be the needed client side for hooking 9p servers into the file system name space.

I'm hoping we can see v9fs in the kernel someday.

thanks

ron

#55 lore raw
In reply to Andrew Morton
Quoted 4 lines · Andrew Morton

vm-early-zone-reclaim

Needs some convincing benchmark numbers to back it up. Otherwise OK.

The only benchmarks I have for this were included in my last mail to linux-mm:

http://marc.theaimsgroup.com/?l=linux-mm&m=111763597218177&w=2

Are they convincing? Well, the patch doesn't seem to make the memory thrashing case much worse ("make -j" kernbench run) which is a good thing since the VM is trying to reclaim much earlier.

In the same e-mail I mention that there is a fairly good performance gain in the optimal case, where processes are tied to a single node and the node's memory is filled with page cache. With zone reclaim turned on the "make -j8" kernel build runs in 700 seconds; 735 seconds with no reclaim.

mh

Signature · 2 lines
Martin Hicks                Wild Open Source Inc.
[email protected]     613-266-2296
#56 lore raw
Quoted 16 lines · Martin Hicks
Quoted 4 lines · Andrew Morton

vm-early-zone-reclaim

Needs some convincing benchmark numbers to back it up. Otherwise OK.

The only benchmarks I have for this were included in my last mail to linux-mm:

http://marc.theaimsgroup.com/?l=linux-mm&m=111763597218177&w=2

Are they convincing? Well, the patch doesn't seem to make the memory thrashing case much worse ("make -j" kernbench run) which is a good thing since the VM is trying to reclaim much earlier.

Quoted 5 lines

In the same e-mail I mention that there is a fairly good performance gain in the optimal case, where processes are tied to a single node and the node's memory is filled with page cache. With zone reclaim turned on the "make -j8" kernel build runs in 700 seconds; 735 seconds with no reclaim.

Ah, OK, I failed to capture that info. (I always have to move the info in the [patch 0/n] email into the first real patch, and this time I didn't)

Thanks.

#57 lore raw
Quoted 3 lines · Andrew Morton

Ah, OK, I failed to capture that info. (I always have to move the info in the [patch 0/n] email into the first real patch, and this time I didn't)

Oops. I'll try to remember to stick the benchmark info into one of the real patches next time.

mh

Signature · 2 lines
Martin Hicks                Wild Open Source Inc.
[email protected]     613-266-2296
#58 lore raw

Re: -mm -> 2.6.13 merge status (wireless)

In reply to Andrew Morton

Hi!

Quoted 4 lines

This summarises my current thinking on various patches which are presently in -mm. I cover large things and small-but-controversial things. Anything which isn't covered here (and that's a lot of material) is probably a "will merge", unless it obviously isn't.

I'd like to ask about 802.11 stack and ipw2100 in particular... Is it "small enough that it did not need mentioning"?
Working wireless in mainline would be great...
Pavel

Signature · 1 line
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         
#59 lore raw

Re: -mm -> 2.6.13 merge status (wireless)

Quoted 11 lines · Pavel Machek

Hi!

Quoted 4 lines

This summarises my current thinking on various patches which are presently in -mm. I cover large things and small-but-controversial things. Anything which isn't covered here (and that's a lot of material) is probably a "will merge", unless it obviously isn't.

I'd like to ask about 802.11 stack and ipw2100 in particular... Is it "small enough that it did not need mentioning"?
Working wireless in mainline would be great...

That's up to Jeff.

#60 lore raw

Re: -mm -> 2.6.13 merge status (wireless)

Quoted 10 lines · Andrew Morton
Quoted 8 lines · Pavel Machek

Hi!

Quoted 4 lines

This summarises my current thinking on various patches which are presently in -mm. I cover large things and small-but-controversial things. Anything which isn't covered here (and that's a lot of material) is probably a "will merge", unless it obviously isn't.

Quoted 6 lines
Quoted 3 lines

I'd like to ask about 802.11 stack and ipw2100 in particular... Is it "small enough that it did not need mentioning"?
Working wireless in mainline would be great...

That's up to Jeff.

802.11 stack is still too ipw-specific.

Someone needs to get together another driver using 802.11 stack (such as HostAP, the original location of much of the code).

So, the merge criteria is: something other than ipw uses it.

Otherwise, it'll never be generic...

Jeff, who has several SuSE wireless patches to merge still

#61 lore raw
In reply to Andrew Morton
Quoted 4 lines · Andrew Morton

git-ocfs

The OCFS2 filesystem. OK by me, although I'm not sure it's had enough review.

Every time I come up with a complaint about ocfs2, the Oracle guys manage to shoot me down. I think it's OK to merge.

Quoted 4 lines

sparsemem

OK by me for a merge. Need to poke arch maintainers first, check that they've looked at it sufficiently closely.

seems sane, though there are some whitespace niggles that should be cleaned up

Quoted 3 lines

rapidio-*

Will merge.

send through netdev, as is proper

Quoted 7 lines

dlm-*.patch: Red Hat distributed lock manager

Hard. Right now it seems that no in-kernel projects will use this and only one out-of-kernel project will use it. Shelve the problem until after Kernel Summit, where some light may be shed.

Opinions are sought...

I hate to say it, since its my own employer, but I agree: wait for in-kernel users, at the very least.

Quoted 8 lines

connector.patch

Nice idea IMO, but there are still questions around the implementation. More dialogue needed ;)

connector-add-a-fork-connector.patch

OK, but needs connector.

I don't like connector

Quoted 4 lines

inotify

There are still concerns about the userspace API and internal implementation details. More slogging needed.

We should ask hpa what he needs for kernel.org. Ideally kernel.org probably wants <something> that facilitates listening to <something> for a list of files being changed. That would greatly speed up the robots, and possibly rsync-like activities too.

Quoted 4 lines

pcmcia-*.patch

Makes the pcmcia layer generate hotplug events and deprecates cardmgr. Will merge.

Testing? The goal behind the patch is certainly good, but I worry about exposure.

Quoted 12 lines

cachefs

This is a ton of code which knows rather a lot about pagecache internals. It allows the AFS client to cache file contents on a local blockdev.

I don't think it's a justified addition for only AFS and I'd prefer to see it proven for NFS as well.

Issues around add-page-becoming-writable-notification.patch need to be resolved.

Quoted 6 lines

cachefs-for-nfs

A recent addition. Needs review from NFS developers and considerably more testing.

These things aren't looking likely for 2.6.13.

If I could vote more than once, I would! I really like cachefs, and have been pushing for its inclusion for a while.

Quoted 4 lines

kexec and kdump

I guess we should merge these.

Quoted 8 lines

I'm still concerned that the various device shutdown problems will mean that the success rate for crashing kernels is not high enough for kdump to be considered a success. In which case in six months time we'll hear rumours about vendors shipping wholly different crashdump implementations, which would be quite bad.

But I think this has gone as far as it can go in -mm, so it's a bit of a punt.

I'm not particularly pleased with these, and indeed vendors ARE shipping other crashdump methods.

Quoted 7 lines

reiser4

Merge it, I guess.

The patches still contain all the reiser4-specific namespace enhancements, only it is disabled, so it is effectively dead code. Maybe we should ask that it actually be removed?

The plugin stuff is crap. This is not a filesystem but a filesystem + new layer. IMO considered in that light, it duplicates functionality elsewhere.

Quoted 4 lines

v9fs

I'm not sure that this has a sufficiently high usefulness-to-maintenance-cost ratio.

agreed (though I think 9P is neat)

Quoted 3 lines

It has been said that a userspace NFS server can be used to get full NFS server functionality with FUSE. I think the half-assed kernel implementation should be done away with.

"It has been said" -- its true. A userspace NFS server can do 100% of userspace FS functionality.

Jeff

#62 lore raw
Quoted 9 lines · Jeff Garzik
Quoted 4 lines

inotify

There are still concerns about the userspace API and internal implementation details. More slogging needed.

We should ask hpa what he needs for kernel.org. Ideally kernel.org probably wants <something> that facilitates listening to <something> for a list of files being changed. That would greatly speed up the robots, and possibly rsync-like activities too.

I've talked to some people who've hooked inotify into rsync successfully. Cool hack.

Robert Love

#63 lore raw
Quoted 7 lines · Robert Love
Quoted 4 lines

We should ask hpa what he needs for kernel.org. Ideally kernel.org probably wants <something> that facilitates listening to <something> for a list of files being changed. That would greatly speed up the robots, and possibly rsync-like activities too.

I've talked to some people who've hooked inotify into rsync successfully. Cool hack.

I noticed that select() is not working on real files. Could inotify be used to fix select()?

#64 lore raw
Quoted 2 lines · Christoph Lameter

I noticed that select() is not working on real files. Could inotify be used to fix select()?

Select the system call? It should work fine. ;-)

Who is confused?

Robert Love

#65 lore raw
Quoted 6 lines · Robert Love
Quoted 2 lines · Christoph Lameter

I noticed that select() is not working on real files. Could inotify be used to fix select()?

Select the system call? It should work fine. ;-)

Hmmm. I just wrote an app that uses select to do essentially a "tail" waiting for new content in a log file. The file descriptors for real disk files are always ready even if there is no content available for the application.

The file is positioned at the end of the file after open via lseek. select tells me that data is available but the read() returns zero bytes.

The current fix on the app level is to checking if useful work was done as a result of "READY" file descriptors. If the read() operations do not return any data then the app will simply sleep for a couple of seconds. So the app degenerates to a kind of poll mode if disk files are used.

#66 lore raw
In reply to Robert Love
Quoted 10 lines · Robert Love
Quoted 2 lines · Christoph Lameter

I noticed that select() is not working on real files. Could inotify be used to fix select()?

Select the system call? It should work fine. ;-)

Who is confused?

Robert Love

Sounds interesting. tail -f could use it. Instead of sleep 1, seek to current position, read to eof; just select() for read on the file and sleep in select() until someone else writes to that file.

I've never tried doing that. It might work, for all I know.

Signature · 1 line
Zan Lynx <[email protected]>
#67 lore raw
Quoted 1 line · Zan Lynx

I've never tried doing that. It might work, for all I know.

I was told that Linux cannot do this. It always returns the filehandles as ready for disk files.

#68 lore raw
Quoted 6 lines · Christoph Lameter
Quoted 1 line · Zan Lynx

I've never tried doing that. It might work, for all I know.

I was told that Linux cannot do this. It always returns the filehandles as ready for disk files.

Inotify would definitely work.

Robert Love

#69 lore raw
Quoted 4 lines · Robert Love
Quoted 2 lines

I was told that Linux cannot do this. It always returns the filehandles as ready for disk files.

Inotify would definitely work.

Well we could use it in kernel to make select() work correctly. For disk files set up a notification for write and then only return from select if new data is available.

#70 lore raw
Quoted 10 lines · Christoph Lameter
Quoted 4 lines · Robert Love
Quoted 2 lines

I was told that Linux cannot do this. It always returns the filehandles as ready for disk files.

Inotify would definitely work.

Well we could use it in kernel to make select() work correctly. For disk files set up a notification for write and then only return from select if new data is available.

You could do it inside glibc.

Signature · 1 line
Zan Lynx <[email protected]>
#71 lore raw
In reply to Christoph Lameter
Quoted 1 line · Christoph Lameter

Well we could use it in kernel to make select() work correctly.

select() already works correctly. It answers the "will I not block if I call read()/write() on this" question, and since you never block on files (assuming infinite disk speed ;-) select() will always return True on it.

You can't change this, it's in POSIX.

... or maybe I misunderstood your comment.

Signature · 6 lines
Matthias Urlichs   |   {M:U} IT Design @ m-u-it.de   |  [email protected]
Disclaimer: The quote was selected randomly. Really. | http://smurf.noris.de
 - -
"I don't really miss God
 but i sure miss Santa Claus!"
      [Courtney Love]
#72 lore raw
In reply to Christoph Lameter
Quoted 6 lines · Christoph Lameter
Quoted 1 line · Zan Lynx

I've never tried doing that. It might work, for all I know.

I was told that Linux cannot do this. It always returns the filehandles as ready for disk files.

That is because disk files are always ready - select/poll are for waits for data (or space) to become available not for events in the sense of inotify.
That said there *is* scope in the poll() API [but not select()] to add a new kind of poll notification type.

#73 lore raw
In reply to Zan Lynx
Quoted 10 lines · On Tue, 21 Jun 2005 14:02:09 -0600 Zan Lynx <[email protected]> wrote:
Quoted 9 lines · Robert Love
Quoted 2 lines · Christoph Lameter

I noticed that select() is not working on real files. Could inotify be used to fix select()?

Select the system call? It should work fine. ;-)

Who is confused?

Quoted 9 lines
Quoted 1 line

Robert Love

Sounds interesting. tail -f could use it. Instead of sleep 1, seek to current position, read to eof; just select() for read on the file and sleep in select() until someone else writes to that file.

I've never tried doing that. It might work, for all I know.

Signature · 1 line
Zan Lynx <[email protected]>

Posix requires select() of regular files always return true:
http://www.opengroup.org/onlinepubs/009695399/functions/select.html

File descriptors associated with regular files shall always select true for ready to read, ready to write, and error conditions.

Signature · 1 line
Stephen Hemminger	<[email protected]>
#74 lore raw
In reply to Christoph Lameter
Quoted 10 lines · Christoph Lameter
Quoted 4 lines · Robert Love
Quoted 1 line

We should ask hpa what he needs for kernel.org. Ideally kernel.org probably wants <something> that facilitates listening to <something> for a list of files being changed. That would greatly speed up the robots, and possibly rsync-like activities too.

I've talked to some people who've hooked inotify into rsync successfully. Cool hack.

I noticed that select() is not working on real files. Could inotify be used to fix select()?

Non-blocking file I/O is an open issue.

AIO is probably a better path.

Jeff

#75 lore raw
Quoted 3 lines · Jeff Garzik

Non-blocking file I/O is an open issue.

AIO is probably a better path.

AIO is requiring you to poll and check if I/O is complete. select() does not require any polling and just needs to be made to work the way it was intended to.

#76 lore raw
Quoted 9 lines · Christoph Lameter
Quoted 3 lines · Jeff Garzik

Non-blocking file I/O is an open issue.

AIO is probably a better path.

AIO is requiring you to poll and check if I/O is complete. select() does

Incorrect. The entire point of AIO is that its an async callback system, when the I/O is complete... just like the kernel's internal I/O request queue system.

Jeff

#77 lore raw
Quoted 5 lines · Jeff Garzik
Quoted 1 line

AIO is requiring you to poll and check if I/O is complete. select() does

Incorrect. The entire point of AIO is that its an async callback system, when the I/O is complete... just like the kernel's internal I/O request queue system.

Hmmm.. Okay it may work like dnotify. You get some signal and then its up to you to figure out what was going on. Traditionally select() does that all for you and tells you which stream got input.

#78 lore raw
In reply to Jeff Garzik
Quoted 6 lines · Jeff Garzik
Quoted 3 lines · Andrew Morton

rapidio-*

Will merge.

send through netdev, as is proper

rapidio-support-net-driver.patch is the only netdev portion.

-Matt

#79 lore raw
In reply to Jeff Garzik

On Tue, 21 Jun 2005 11:26:44 -0400 Jeff Garzik wrote:

| Andrew Morton wrote:
| 
| > connector.patch
| > 
| >     Nice idea IMO, but there are still questions around the
| >     implementation.  More dialogue needed ;)
| > 
| > connector-add-a-fork-connector.patch
| > 
| >     OK, but needs connector.
| 
| I don't like connector

can you be more specific, like you did with reiser4?

| > kexec and kdump
| > 
| >     I guess we should merge these.
| > 
| >     I'm still concerned that the various device shutdown problems will
| >     mean that the success rate for crashing kernels is not high enough for
| >     kdump to be considered a success.  In which case in six months time we'll
| >     hear rumours about vendors shipping wholly different crashdump
| >     implementations, which would be quite bad.
| > 
| >     But I think this has gone as far as it can go in -mm, so it's a bit of
| >     a punt.
| 
| I'm not particularly pleased with these, and indeed vendors ARE shipping 
| other crashdump methods.

any specifics on the "not particularly pleased" part?

| > reiser4
| > 
| >     Merge it, I guess.
| > 
| >     The patches still contain all the reiser4-specific namespace
| >     enhancements, only it is disabled, so it is effectively dead code.  Maybe
| >     we should ask that it actually be removed?
| 
| The plugin stuff is crap.  This is not a filesystem but a filesystem + 
| new layer.  IMO considered in that light, it duplicates functionality 
| elsewhere.

I don't think that r4 is just a filesystem either, but you know more about that than I do.

thanks,
---
~Randy

#80 lore raw
In reply to Jeff Garzik
Quoted 10 lines · Jeff Garzik
Quoted 5 lines

reiser4

The plugin stuff is crap. This is not a filesystem but a filesystem + new layer. IMO considered in that light, it duplicates functionality elsewhere.

What functionality where? Please remember that this is per file, per item, per node, per attribute, per disk format, per bitmap, per super block, etc., abstracting, not per filesystem abstracting.

Plugins allow a number of things:

1) They allow us to never pay the cost to change the disk format again, no matter how much we add in future years. This really matters: the prohibitive cost of disk format changes are the number one impediment to filesystem improvements, and the primary reason why most filesystems ossify after time has past.

2) They allow us to easily structure code for reuse. If we want to
create a new kind of file that is like some other kind of file except for one thing, we just write the one thing, and then easily reuse all the other code, and create a new plugin id.

The use of plugins forced all the programmers to think about reusability at every layer of design. V3 of reiserfs is way too hard to work on and modify. If you ask one of the team to code something for V3 instead of V4, they quietly groan at the thought. It is just so much easier to do in V4.

When I asked one of our team to completely change the key assignment algorithm for V4 (which controls what things get packed near what in the tree), he complained that it would take 6 weeks to do it. Under V3 it would have taken 3 months. It took him 3 days, and now to change it again would take him 3 hours I think. Oh, by the way, the change boosted performance dramatically.

If you take the time to become familiar with coding within our plugin layer, I think you will find yourself wanting the same to exist for other filesystems. Of course, no other filesystem needs to be impacted by our plugin layer, and there is no way reiser4 could easily be rewritten to exist without it (it would be like requiring that the kernel not use function calls and only use gotos).

Reiser4's plugin layer has as its explicit objective making it possible for the weekend hacker to add something useful to reiser4 and send it in for inclusion. We want to democratize filesystem innovation so that random bright guys who usually work on something other than filesystems can act on their bright ideas without spending 3 years in the filesystem field to do it. I believe that most great filesystem innovations are envisioned by persons not working on filesystems, and go nowhere because of the especially high cost of entry into our field.

I am not as bright as other filesystem developers, and so we must tinker with three ideas and keep one of them. The speed of tinkering is crucial to us, and the plugin layer increases that speed several fold.

Please reconsider your view.

#81 lore raw

Hans, we had this discussion before. And the consensus was pretty simple:
the disk structure plugins are your business and fine to keep. The higher-level pluging that just add another useless abstraction below file_operation/inode_operation/etc. are not. keep the former and kill the latter and you're one step further.

#82 lore raw

reiser4 plugins

Christoph,

Reiser4 users love the plugin concept, and all audiences which have listened to a presentation on plugins have been quite positive about it. Many users think it is the best thing about reiser4. Can you articulate why you are opposed to plugins in more detail? Perhaps you are simply not as familiar with it as the audiences I have presented to. Perhaps persons on our mailing list can comment.....

In particular, what is wrong with having a plugin id associated with every file, storing the pluginid on disk in permanent storage in the stat data, and having that plugin id define the set of methods that implement the vfs operations associated with a particular file, rather than defining VFS methods only at filesystem granularity?

What is wrong with having an encryption plugin implemented in this manner? What is wrong with being able to have some files implemented using a compression plugin, and others in the same filesystem not.

What is wrong with having one file in the FS use a write only plugin, in which the encrypion key is changed with every append in a forward but not backward computable manner, and in order to read a file you must either have a key that is stored on another computer or be reading what was written after the moment of cracking root?

What is wrong with having a set of critical data files use a CRC checking file plugin?

What we have hurts no one but us. I have never seen an audience for one of my talks that thought it hurt us..... most audiences think it is great.

Let us tinker with our FS, and you tinker with yours, and so long as what we do does not affect your FS, let the users choose.

In the end, somebody will write a new fs that steals the good ideas from both of us, and obsoletes us both. They can only do this though if we are left to be both free to implement differing filesystem designs.

I do not tell you how to design XFS, why are you making my life unpleasant?

Quoted 9 lines · Christoph Hellwig

Hans, we had this discussion before. And the consensus was pretty simple:
the disk structure plugins are your business and fine to keep. The higher-level pluging that just add another useless abstraction below file_operation/inode_operation/etc. are not. keep the former and kill the latter and you're one step further.

#83 lore raw

Re: reiser4 plugins

Quoted 9 lines · Hans Reiser

Christoph,

Reiser4 users love the plugin concept, and all audiences which have listened to a presentation on plugins have been quite positive about it. Many users think it is the best thing about reiser4. Can you articulate why you are opposed to plugins in more detail? Perhaps you are simply not as familiar with it as the audiences I have presented to. Perhaps persons on our mailing list can comment.....

Quoted 5 lines

In particular, what is wrong with having a plugin id associated with every file, storing the pluginid on disk in permanent storage in the stat data, and having that plugin id define the set of methods that implement the vfs operations associated with a particular file, rather than defining VFS methods only at filesystem granularity?

You're basically implementing another VFS layer inside of reiser4, which is a big layering violation.

This sort of feature should -not- be done at the low-level filesystem level.

What happens if people decide plugins are a good idea, and they want them in ext3? We need massive surgery to extract the guts from reiser4.

Jeff

#84 lore raw

Re: reiser4 plugins

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 18 lines · Jeff Garzik
Quoted 9 lines · Hans Reiser

Christoph,

Reiser4 users love the plugin concept, and all audiences which have listened to a presentation on plugins have been quite positive about it. Many users think it is the best thing about reiser4. Can you articulate why you are opposed to plugins in more detail? Perhaps you are simply not as familiar with it as the audiences I have presented to. Perhaps persons on our mailing list can comment.....

Quoted 5 lines

In particular, what is wrong with having a plugin id associated with every file, storing the pluginid on disk in permanent storage in the stat data, and having that plugin id define the set of methods that implement the vfs operations associated with a particular file, rather than defining VFS methods only at filesystem granularity?

Quoted 2 lines

You're basically implementing another VFS layer inside of reiser4, which is a big layering violation.

There's been sloppy code in the kernel before. I remember one bit in particular which was commented "Fuck me gently with a chainsaw." If I remember correctly, this had all of the PCI ids and the names and manufacturers of the corresponding devices -- in a data structure -- in C source code. It was something like one massive array definition, or maybe it was a structure. I don't remember exactly, but...

The point is, this was in the kernel for quite awhile, and it was so ugly that someone would rather be fucked with a chainsaw. If something that bad can make it in the kernel and stay for awhile because it worked, and no one wanted to replace it -- nowdays, that database is kept in userland as some nice text format -- then I vote for putting Reiser4 in the kernel now, and ironing the sloppiness ("violation") out later. It runs now.

Quoted 2 lines

This sort of feature should -not- be done at the low-level filesystem level.

I agree there, too. In fact, some people have suggested that all "legacy" (read: non-reiser) filesystems should be implemented as Reiser4 plugins, effectively killing VFS.*

So, Reiser4 may eventually take over VFS and be the only Linux filesystem, but if so, it will have to do it much more slowly. Take the good ideas -- things like plugins -- and make them at least look like incremental updates to the current VFS, and make them available to all filesystems.

Eventually, this would result in a full merge of Reiser and Linux, such that the only thing left of "Reiser4" are a few plugins -- things like storage plugins and maybe some more exotic things like fibration that I don't really understand.

Quoted 2 lines

What happens if people decide plugins are a good idea, and they want them in ext3? We need massive surgery to extract the guts from reiser4.

And here is the crucial point. Reiser4 is usable and useful NOW, not after it has undergone massive surgery, and Namesys is bankrupt, and users have given up and moved on to XFS. But the massive surgery should happen eventually, partly to make all filesystems better (see below), and partly to make the transition easier and more palatable for those who don't work for Namesys.

* Imagine ext3 as a storage-level plugin for reiser4. You get one benefit immediately -- lazy allocation. Lazy allocation is nice both for fragmentation and for busy systems writing and nuking a lot of temporary files. Imagine a file which is written and then deleted before it hits disk -- it should never touch the disk, regardless of the underlying storage layout.

Other benefits are subtler but inevitable. Right now, it would be an understatement to say that there's duplication of effort between ext3, xfs, and reiser4. And yet, I don't think there are many core design decisions that influence my decision as to which I pick up. I just want it to be fast, stable, and have a stable on-disk format so I don't have to reformat too often. I honestly don't care about how XFS is segmenting the disk, or Reiser4 packs really well, or ext3 can be read as ext2 and flushes to disk every five seconds. These are the kinds of things which should be set to sane defaults, tunable for enterprise users, but are not really enough to warrant completely separate code bases.

I would imagine that it wouldn't be too long after this before an uber-fs rose, something which combined enough of the strong points of all the existing Linux filesystems that no one would use anything else.
But, Linux still needs support for FAT32, ISO9660, UDF, and all the
other filesystems which won't go away as easily as XFS and ext3, and it would be nice if these could all share as much code as possible.

I don't know if storage plugins really work that way, but they should.

I think. I don't work here.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrjoNHgHNmZLgCUhAQIYYw/7BWZ0gVvy0ln5tRo405yUoRHJ/jVFBGyP 5pQ610ECMZORVWRO1qP/NXbvGZwDjEggM5iIeahsGqnBWzNGDsB6RslMUZAxzCYy iAovi0881zoARf3dmhKrDgbGkvNLPTx+/ypa20oRcHLnyI+NAjerUxNuvGc79PJn Ybm9JhX6tQsqGIKjZye9uNDHCifLqzA1gdxucPwWo9sz4ymzM9FgsMGvb+IxrU50 2a2G2K6AHcH+nkomEHw3xgY3PmUZUy0s2hQDSkJx6dCido7fGZwwykaSXm4IZs9s xZqBxKx32G/LDnDV3W8gpj8ZisUE+58kefRbIjo4Ml6IzgXvQqpRjaQOuz8JoKDX 9KUV43tcZkPpK+neIYPQYpXCrdMQqB5+ISpbNKuz/Z/abkcVw1042sy0EKM+/VnD n3Jr7PpSyk0lfCyADr+33qnqPQRFq02gQTohg9FheoMthhV01aaeGW5ExmTM/Wwa 6Dmv/qnn2oNImi+Ebz5u3Lbnz7lL3MVpRL4aoMmEOVUB3xhehnxesf//yacBmwj9 M/3KVae9epwX4K6yi8qQXzH4160IBFHpWUxBLc9LnOZlHQuZ+Fz3BIrbKQBvmaHT 7lrwi0Os6TmiBGMSFd6OUWHcYs4p97aMw30NG33fCL6e8X6oNVFwwnJXZpBPN1Mr +lrsVAywKEI=
=rHdK
-----END PGP SIGNATURE-----

#85 lore raw

Re: reiser4 plugins

First of all, I'm HW engineer, and don't know much about implementation details of FS. I know some coding, I've coded for different levels from linux device drivers to GUI's, but I know a lot about integration between levels of hierarchy.

I generally agree with the idea that high level functionality such as compressing, encrypting, ...etc should not be implemented in the FS. But, the term "high-level" is a relative term and three years down the road, what we consider high-level now becomes low-level.

my suggestion is to keep everything as is. if Hans and the reiser4 guys want to implement high level stuff as plug-ins that's fine. If the VFS guys want to implement the same functionality in VFS on top of the system FS; they can do it too.

BUT PLEASE MAKE THE API COMPATIBLE!!!!

programmers on both sides should discuss the features and make sure the API is compatible.

in such a scenario, if I want to compress a directory called /mystuff/data on a reiser4 FS, I could call the VFS API to do so; The VFS finds that the system is reiser4 and compression is supported, so it passes the call to reiser4. On another hand, if the system is ext3 and it doens't support compression, VFS performs the task and passes the file to ext3.

My point is that the effect of gravity is upon us, high level stuff will become low-level in a short time. Moreover, at the low level in FS and especially when done by FS guys, functions such as compression can be done more efficiently than in VFS space.

my worries are that the waste of effort by implementing the same functionality in two levels in the hierarchy, and incompatible API can cause serious headache.

just my two cents.

-Bedros

Quoted 123 lines · David Masover

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 18 lines · Jeff Garzik
Quoted 9 lines · Hans Reiser

Christoph,

Reiser4 users love the plugin concept, and all audiences which have listened to a presentation on plugins have been quite positive about it. Many users think it is the best thing about reiser4. Can you articulate why you are opposed to plugins in more detail? Perhaps you are simply not as familiar with it as the audiences I have presented to. Perhaps persons on our mailing list can comment.....

Quoted 5 lines

In particular, what is wrong with having a plugin id associated with every file, storing the pluginid on disk in permanent storage in the stat data, and having that plugin id define the set of methods that implement the vfs operations associated with a particular file, rather than defining VFS methods only at filesystem granularity?

Quoted 2 lines

You're basically implementing another VFS layer inside of reiser4, which is a big layering violation.

There's been sloppy code in the kernel before. I remember one bit in particular which was commented "Fuck me gently with a chainsaw." If I remember correctly, this had all of the PCI ids and the names and manufacturers of the corresponding devices -- in a data structure -- in C source code. It was something like one massive array definition, or maybe it was a structure. I don't remember exactly, but...

The point is, this was in the kernel for quite awhile, and it was so ugly that someone would rather be fucked with a chainsaw. If something that bad can make it in the kernel and stay for awhile because it worked, and no one wanted to replace it -- nowdays, that database is kept in userland as some nice text format -- then I vote for putting Reiser4 in the kernel now, and ironing the sloppiness ("violation") out later. It runs now.

Quoted 2 lines

This sort of feature should -not- be done at the low-level filesystem level.

I agree there, too. In fact, some people have suggested that all "legacy" (read: non-reiser) filesystems should be implemented as Reiser4 plugins, effectively killing VFS.*

So, Reiser4 may eventually take over VFS and be the only Linux filesystem, but if so, it will have to do it much more slowly. Take the good ideas -- things like plugins -- and make them at least look like incremental updates to the current VFS, and make them available to all filesystems.

Eventually, this would result in a full merge of Reiser and Linux, such that the only thing left of "Reiser4" are a few plugins -- things like storage plugins and maybe some more exotic things like fibration that I don't really understand.

Quoted 2 lines

What happens if people decide plugins are a good idea, and they want them in ext3? We need massive surgery to extract the guts from reiser4.

And here is the crucial point. Reiser4 is usable and useful NOW, not after it has undergone massive surgery, and Namesys is bankrupt, and users have given up and moved on to XFS. But the massive surgery should happen eventually, partly to make all filesystems better (see below), and partly to make the transition easier and more palatable for those who don't work for Namesys.

* Imagine ext3 as a storage-level plugin for reiser4. You get one benefit immediately -- lazy allocation. Lazy allocation is nice both for fragmentation and for busy systems writing and nuking a lot of temporary files. Imagine a file which is written and then deleted before it hits disk -- it should never touch the disk, regardless of the underlying storage layout.

Other benefits are subtler but inevitable. Right now, it would be an understatement to say that there's duplication of effort between ext3, xfs, and reiser4. And yet, I don't think there are many core design decisions that influence my decision as to which I pick up. I just want it to be fast, stable, and have a stable on-disk format so I don't have to reformat too often. I honestly don't care about how XFS is segmenting the disk, or Reiser4 packs really well, or ext3 can be read as ext2 and flushes to disk every five seconds. These are the kinds of things which should be set to sane defaults, tunable for enterprise users, but are not really enough to warrant completely separate code bases.

I would imagine that it wouldn't be too long after this before an uber-fs rose, something which combined enough of the strong points of all the existing Linux filesystems that no one would use anything else. But, Linux still needs support for FAT32, ISO9660, UDF, and all the other filesystems which won't go away as easily as XFS and ext3, and it would be nice if these could all share as much code as possible.

I don't know if storage plugins really work that way, but they should.

I think. I don't work here.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrjoNHgHNmZLgCUhAQIYYw/7BWZ0gVvy0ln5tRo405yUoRHJ/jVFBGyP 5pQ610ECMZORVWRO1qP/NXbvGZwDjEggM5iIeahsGqnBWzNGDsB6RslMUZAxzCYy iAovi0881zoARf3dmhKrDgbGkvNLPTx+/ypa20oRcHLnyI+NAjerUxNuvGc79PJn Ybm9JhX6tQsqGIKjZye9uNDHCifLqzA1gdxucPwWo9sz4ymzM9FgsMGvb+IxrU50 2a2G2K6AHcH+nkomEHw3xgY3PmUZUy0s2hQDSkJx6dCido7fGZwwykaSXm4IZs9s xZqBxKx32G/LDnDV3W8gpj8ZisUE+58kefRbIjo4Ml6IzgXvQqpRjaQOuz8JoKDX 9KUV43tcZkPpK+neIYPQYpXCrdMQqB5+ISpbNKuz/Z/abkcVw1042sy0EKM+/VnD n3Jr7PpSyk0lfCyADr+33qnqPQRFq02gQTohg9FheoMthhV01aaeGW5ExmTM/Wwa 6Dmv/qnn2oNImi+Ebz5u3Lbnz7lL3MVpRL4aoMmEOVUB3xhehnxesf//yacBmwj9 M/3KVae9epwX4K6yi8qQXzH4160IBFHpWUxBLc9LnOZlHQuZ+Fz3BIrbKQBvmaHT 7lrwi0Os6TmiBGMSFd6OUWHcYs4p97aMw30NG33fCL6e8X6oNVFwwnJXZpBPN1Mr +lrsVAywKEI=
=rHdK
-----END PGP SIGNATURE-----

#86 lore raw

Re: reiser4 plugins

In reply to David Masover
Quoted 7 lines · David Masover

There's been sloppy code in the kernel before. I remember one bit in particular which was commented "Fuck me gently with a chainsaw." If I remember correctly, this had all of the PCI ids and the names and manufacturers of the corresponding devices -- in a data structure -- in C source code. It was something like one massive array definition, or maybe it was a structure. I don't remember exactly, but...

Quoted 7 lines

The point is, this was in the kernel for quite awhile, and it was so ugly that someone would rather be fucked with a chainsaw. If something that bad can make it in the kernel and stay for awhile because it worked, and no one wanted to replace it -- nowdays, that database is kept in userland as some nice text format -- then I vote for putting Reiser4 in the kernel now, and ironing the sloppiness ("violation") out later. It runs now.

Existence of ugly code is not an excuse to add more.

We have to maintain said ugly code for decades. Maintainability is a big deal when you deal with the timeframes we deal with.

Quoted 5 lines

So, Reiser4 may eventually take over VFS and be the only Linux filesystem, but if so, it will have to do it much more slowly. Take the good ideas -- things like plugins -- and make them at least look like incremental updates to the current VFS, and make them available to all filesystems.

This is how all Linux development is done.

The code evolves over time.

You have just described the path ReiserFS needs to take, in order to get this stuff into the kernel, in fact.

Quoted 6 lines

And here is the crucial point. Reiser4 is usable and useful NOW, not after it has undergone massive surgery, and Namesys is bankrupt, and users have given up and moved on to XFS. But the massive surgery should happen eventually, partly to make all filesystems better (see below), and partly to make the transition easier and more palatable for those who don't work for Namesys.

We care about technical merit, not some random company's financial solvancy. Reiser4 has layering violations, and doesn't work with the current security layer. Those are two biggies.

There is no entitlement to be merged in the upstream kernel. If people don't like how the Linux kernel is managed, they are free to maintain their own fork. If its better, people will use that instead.

Quoted 9 lines

I would imagine that it wouldn't be too long after this before an uber-fs rose, something which combined enough of the strong points of all the existing Linux filesystems that no one would use anything else.
But, Linux still needs support for FAT32, ISO9660, UDF, and all the
other filesystems which won't go away as easily as XFS and ext3, and it would be nice if these could all share as much code as possible.

I don't know if storage plugins really work that way, but they should.

No, they shouldn't.

Quoted 1 line

I think. I don't work here.

Obviously.

Jeff

#87 lore raw

Re: reiser4 plugins

Quoted 6 lines · Jeff Garzik
Quoted 5 lines

after it has undergone massive surgery, and Namesys is bankrupt, and users have given up and moved on to XFS. But the massive surgery should happen eventually, partly to make all filesystems better (see below), and partly to make the transition easier and more palatable for those who don't work for Namesys.

Quoted 7 lines

And here is the crucial point. Reiser4 is usable and useful NOW, not

We care about technical merit, not some random company's financial solvancy.

Hmm, that must be why Daniel Robbins, the creator of the distro with
the technically superior Gentoo approach, is still with us, yes?

If you cared about technical merit, you would discuss benchmarks, yes?

#88 lore raw

Re: reiser4 plugins

In reply to Jeff Garzik

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 18 lines · Jeff Garzik
Quoted 7 lines · David Masover

There's been sloppy code in the kernel before. I remember one bit in particular which was commented "Fuck me gently with a chainsaw." If I remember correctly, this had all of the PCI ids and the names and manufacturers of the corresponding devices -- in a data structure -- in C source code. It was something like one massive array definition, or maybe it was a structure. I don't remember exactly, but...

Quoted 7 lines

The point is, this was in the kernel for quite awhile, and it was so ugly that someone would rather be fucked with a chainsaw. If something that bad can make it in the kernel and stay for awhile because it worked, and no one wanted to replace it -- nowdays, that database is kept in userland as some nice text format -- then I vote for putting Reiser4 in the kernel now, and ironing the sloppiness ("violation") out later. It runs now.

Quoted 1 line

Existence of ugly code is not an excuse to add more.

Maybe not, but I'm making a point. I'm sure that, at the time, people wanted something that ran. They wanted lspci to work. It was generally assumed that it would be cleaned up later, and it was. Too much later, but it happened, eventually.

I've been reading a bit of history, and the reason Linux got so popular in the first place was the tendency to include stuff that worked and provided a feature people wanted, even if it was ugly. The philosophy would be: choose a good implementation over an ugly one, but choose an ugly one over nothing at all.

Quoted 2 lines

We have to maintain said ugly code for decades. Maintainability is a big deal when you deal with the timeframes we deal with.

Maintainability is like optimization. The maintainability of a non-working program is irrelevant. You'd be right if we already had plugins-in-the-VFS. We don't. The most maintainable solution for plugins-in-the-FS that actually exists is Reiser4, exactly as it is now
- -- because it is the _only_ one that actually exists right now.

Quoted 7 lines
Quoted 5 lines

So, Reiser4 may eventually take over VFS and be the only Linux filesystem, but if so, it will have to do it much more slowly. Take the good ideas -- things like plugins -- and make them at least look like incremental updates to the current VFS, and make them available to all filesystems.

Quoted 6 lines

This is how all Linux development is done.

The code evolves over time.

You have just described the path ReiserFS needs to take, in order to get this stuff into the kernel, in fact.

This is the path it needs to take, long term, for the sanity of everyone. But I don't think it can get there without being included, short term. People will stomp all over any attempts to change the VFS as "unproven" and "unneccesary". Do you think it will be any easier to get this stuff into the kernel that way?

Better, I think, to drop it in, as-is, and move stuff incrementally from the FS to the VFS. That way, there's at least something intermediate for people to use, test, and hammer/beat down, and for maintainers to get more comfortable with the idea and the logistics of this beast in their kernel.

Quoted 11 lines
Quoted 6 lines

And here is the crucial point. Reiser4 is usable and useful NOW, not after it has undergone massive surgery, and Namesys is bankrupt, and users have given up and moved on to XFS. But the massive surgery should happen eventually, partly to make all filesystems better (see below), and partly to make the transition easier and more palatable for those who don't work for Namesys.

We care about technical merit, not some random company's financial solvancy. Reiser4 has layering violations, and doesn't work with the current security layer. Those are two biggies.

Ah, so you mean to say, you care about how well something fits the current model. That has little to do with technical merit.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrkqoHgHNmZLgCUhAQIKixAAnTJTJJpuCLatFp8syccjNnE7WdHlO/zx G1bWuSthCnb7uaVb8buDeAlpArzttoguKKum0NE0khz/FjKw4YUXH4AEsVYGlZaO nBYpw0MVyypNP+hZhEuo1T826frEOb6Z40Y1WZCpYwAZCs9EQQm7TeYSMjhD17Ew PehYwUFUmnv1S7CpYNQvuboKh/1wuUQb6R2thjuCGJVkif8Mn2U20Fhk1/HIgnIr OHoCD8ZgwoqBDPKQ6V26dUX+ZHzQVJX1j/IgLnnitJ9E4quIHNs33lq4S99DWta6 uDS4hkHgFMRemh37sA0FUMeitFsrwNb2b2f0o/X8MpDJmwbdrdg9kxvwfHqqgaz+ Enj0rBXO8E+5a4STTk4L2LaSR2+knSEFdj53MYYq4ABL07hEbJp2cNFKh5AFXvg0 wg5WoHt4HhhOeuftIG9Twv5tHIC5qoM57ae9yZzj783G9ZnXy0xBefUmH+pWVQsp H1IpKIR4a0l8gV1AkJa6BUyAyzDDObFzmqcZ61W15Y2relD9Ow2qzVqMxroB78uJ O+on741BecGtohB5xdfth9rwDY6JPkDug3C6zHzxSAkGGEnWIn6O8CzcGrGqS0Ta EmB4LGw/fZqGcEYOOErqMC6GuImv2JbjtBOx8nAxM2OhGXFoDiD9FQaDaxWw9zjj ROODOhm0aTA=
=ivqd
-----END PGP SIGNATURE-----

#89 lore raw

Re: reiser4 plugins

David Masover writes:

[...]

Quoted 6 lines

Maintainability is like optimization. The maintainability of a non-working program is irrelevant. You'd be right if we already had plugins-in-the-VFS. We don't. The most maintainable solution for plugins-in-the-FS that actually exists is Reiser4, exactly as it is now
- -- because it is the _only_ one that actually exists right now.

But it is not so. There _are_ plugins-in-the-VFS. VFS operates on opaque objects (inodes, dentries, file system types) through interfaces:
{inode,address_space,dentry,sb,etc.}_operations. Every file system back-end if free to implement whatever number of these interfaces. And the do this already: check the sources; even ext2 does this: e.g., ext2_fast_symlink_inode_operations and ext2_symlink_inode_operations.

This is exactly what upper level reiser4 plugins are for.

I guess that one of Christoph Hellwig's complaints is that reiser4 introduces another layer of abstraction to implement something that already exists.

Nikita.

#90 lore raw

Re: reiser4 plugins

Hello

Quoted 11 lines · Nikita Danilov

David Masover writes:

[...]

Quoted 6 lines

Maintainability is like optimization. The maintainability of a non-working program is irrelevant. You'd be right if we already had plugins-in-the-VFS. We don't. The most maintainable solution for plugins-in-the-FS that actually exists is Reiser4, exactly as it is now
- -- because it is the _only_ one that actually exists right now.

Quoted 7 lines

But it is not so. There _are_ plugins-in-the-VFS. VFS operates on opaque objects (inodes, dentries, file system types) through interfaces:
{inode,address_space,dentry,sb,etc.}_operations. Every file system back-end if free to implement whatever number of these interfaces. And the do this already: check the sources; even ext2 does this: e.g., ext2_fast_symlink_inode_operations and ext2_symlink_inode_operations.

imho, this is something different. Ext2 decides itself how to manage a symlink depending on length of string the symlink is to store. Reiser4 plugins are to allow a user to define himself which operations file is to be managed with.

Quoted 6 lines

This is exactly what upper level reiser4 plugins are for.

I guess that one of Christoph Hellwig's complaints is that reiser4 introduces another layer of abstraction to implement something that already exists.

I do not think that it exists already.
You can have standart type of files and that is all.

Linux filesystem is not supposed to provide anything but plain regular/directory/symlinks/sockets/block device/char device/fifo files.

Existing file API does not allow create anything but that. Merging reiser4 with object plugins would make it necessary to modify VFS layer so that files of arbitrary types could be created.

#91 lore raw

Re: reiser4 plugins

Quoted 22 lines · Vladimir Saveliev

Hello

Quoted 11 lines · Nikita Danilov

David Masover writes:

[...]

Quoted 6 lines

Maintainability is like optimization. The maintainability of a non-working program is irrelevant. You'd be right if we already had plugins-in-the-VFS. We don't. The most maintainable solution for plugins-in-the-FS that actually exists is Reiser4, exactly as it is now
- -- because it is the _only_ one that actually exists right now.

Quoted 7 lines

But it is not so. There _are_ plugins-in-the-VFS. VFS operates on opaque objects (inodes, dentries, file system types) through interfaces:
{inode,address_space,dentry,sb,etc.}_operations. Every file system back-end if free to implement whatever number of these interfaces. And the do this already: check the sources; even ext2 does this: e.g., ext2_fast_symlink_inode_operations and ext2_symlink_inode_operations.

Quoted 4 lines

imho, this is something different. Ext2 decides itself how to manage a symlink depending on length of string the symlink is to store. Reiser4 plugins are to allow a user to define himself which operations file is to be managed with.

I fail to see this as an important distinction:

- ext2 decides what "plugin" to use based on parameters supplied to
the file creation system call;

- reiser4 decides what "plugin" to use based on parameters supplied to the file creation system call.

Quoted 8 lines
Quoted 7 lines

This is exactly what upper level reiser4 plugins are for.

I guess that one of Christoph Hellwig's complaints is that reiser4 introduces another layer of abstraction to implement something that already exists.

Quoted 5 lines

I do not think that it exists already.
You can have standart type of files and that is all.

Linux filesystem is not supposed to provide anything but plain regular/directory/symlinks/sockets/block device/char device/fifo files.

If you are talking about S_IFMT bits they are just a relic to keep user level happy (yes, VFS does few S_ISFOO checks here and there, but they all can be replaced with checks for appropriate operations being non-NULL). Nothing prevents file system from rolling forward whatever exotic objects it wants.

Quoted 5 lines

Existing file API does not allow create anything but that. Merging reiser4 with object plugins would make it necessary to modify VFS layer so that files of arbitrary types could be created.

This is important, but it doesn't depend on whether file system has additional layer if indirection below VFS operations. If file system wants to support extended object types, additional parameters have to be somehow communicated from the user level during object creation. But:

- this is necessary even if file system has no "plugins" below VFS operations;

- this doesn't necessary means changing VFS. For example, these parameters can be set on parent directory (this is how ACLs were integrated into POSIX).

Quoted 1 line

Nikita.

#92 lore raw

Re: reiser4 plugins

In reply to David Masover
Quoted 5 lines · David Masover

I've been reading a bit of history, and the reason Linux got so popular in the first place was the tendency to include stuff that worked and provided a feature people wanted, even if it was ugly. The philosophy would be: choose a good implementation over an ugly one, but choose an ugly one over nothing at all.

And things change over time. Back in those days the linux codebase was small and it was easy to change things all over the place. These times our codebase is huge, and people that know enough parts of the kernel to do big changes are overloaded with work. Thus we have to set our acceptance criteria a lot higher now - else we'd be totally lost with the current size of the project already.

Quoted 8 lines
Quoted 2 lines

We have to maintain said ugly code for decades. Maintainability is a big deal when you deal with the timeframes we deal with.

Maintainability is like optimization. The maintainability of a non-working program is irrelevant. You'd be right if we already had plugins-in-the-VFS. We don't. The most maintainable solution for plugins-in-the-FS that actually exists is Reiser4, exactly as it is now
- -- because it is the _only_ one that actually exists right now.

We do have plugins in the VFS, every filesystem is a set of a few of them and some gluecode.

<skipping a lot stuff>

David and Hans, I've read through my backlog a lot now, and I must say it's pretty pointless - you're discussing lots of highlevel what if and don't actually care about something as boring as actual technical details.

Hans has lots of very skillfull technical people like zam and vs, and maybe he should give them some freedom to sort out technical issues for a basic reiser4 merge, and one that is done we can turn back to discussion of advanced features and their implementation, hopefully with a few more arguments on both sides and a real technical discussion.

#93 lore raw

Re: reiser4 plugins

Quoted 32 lines · Christoph Hellwig
Quoted 5 lines · David Masover

I've been reading a bit of history, and the reason Linux got so popular in the first place was the tendency to include stuff that worked and provided a feature people wanted, even if it was ugly. The philosophy would be: choose a good implementation over an ugly one, but choose an ugly one over nothing at all.

And things change over time. Back in those days the linux codebase was small and it was easy to change things all over the place. These times our codebase is huge, and people that know enough parts of the kernel to do big changes are overloaded with work. Thus we have to set our acceptance criteria a lot higher now - else we'd be totally lost with the current size of the project already.

Quoted 8 lines
Quoted 2 lines

We have to maintain said ugly code for decades. Maintainability is a big deal when you deal with the timeframes we deal with.

Maintainability is like optimization. The maintainability of a non-working program is irrelevant. You'd be right if we already had plugins-in-the-VFS. We don't. The most maintainable solution for plugins-in-the-FS that actually exists is Reiser4, exactly as it is now
- -- because it is the _only_ one that actually exists right now.

We do have plugins in the VFS, every filesystem is a set of a few of them and some gluecode.

<skipping a lot stuff>

David and Hans, I've read through my backlog a lot now, and I must say it's pretty pointless - you're discussing lots of highlevel what if and don't actually care about something as boring as actual technical details.

Quoted 5 lines

Hans has lots of very skillfull technical people like zam and vs, and maybe he should give them some freedom to sort out technical issues for a basic reiser4 merge, and one that is done we can turn back to discussion of advanced features and their implementation, hopefully with a few more arguments on both sides and a real technical discussion.

Unfortunately, this is not only a technical discussion... it is about linux development model too.

Well, about the plugins. We can clean reiser4<->VFS interface up by setting per-vfs-object inode/dentry/super ops instead using of our own dispatcher. So different reiser4 inodes/files will have different i_ops/f_ops. That would be only visible-in-VFS part of reiser4 object plugins.

Would the help to solve "reiser4 plugins" question? It is just as other FS do -- procfs has seq_file and sysconfig interfaces below the VFS and l-k people do not complain each day about layering violation ;-) Procfs is taken as an example because it deals with objects of different types, actually anybody may create own procfs objects more or less general way.

I don't belive that you want to see all reiser4-specific things as item plugins, disk format plugins in the VFS.

Thanks,
Alex.

#94 lore raw

Re: reiser4 plugins

Quoted 4 lines · Alexander Zarochentsev

-- procfs has seq_file and sysconfig interfaces below the VFS and l-k people do not complain each day about layering violation ;-) Procfs is taken as an example because it deals with objects of different types, actually anybody may create own procfs objects more or less general way.

seq_file actually works at the file_operations level, that's exactly what I'm telling you to do. The old sub-callbacks are on their way out.

Quoted 2 lines

I don't belive that you want to see all reiser4-specific things as item plugins, disk format plugins in the VFS.

If you'd read the previous discussions you'd see that no one complained about disk format plugins.

#95 lore raw

Re: reiser4 plugins

Quoted 8 lines · Christoph Hellwig
Quoted 4 lines · Alexander Zarochentsev

-- procfs has seq_file and sysconfig interfaces below the VFS and l-k people do not complain each day about layering violation ;-) Procfs is taken as an example because it deals with objects of different types, actually anybody may create own procfs objects more or less general way.

seq_file actually works at the file_operations level, that's exactly what I'm telling you to do. The old sub-callbacks are on their way out.

not exactly. I meant that seq_file has its own VFS-like thing struct seq_operations.

So I may assume that having own objects and their operations is allowed. The complains are about adding unnecessary level of indirection in the trivial reiser4 wrappers as reiser4_write().

Quoted 5 lines
Quoted 2 lines

I don't belive that you want to see all reiser4-specific things as item plugins, disk format plugins in the VFS.

If you'd read the previous discussions you'd see that no one complained about disk format plugins.

#96 lore raw

Re: reiser4 plugins

Quoted 2 lines · Alexander Zarochentsev

not exactly. I meant that seq_file has its own VFS-like thing struct seq_operations.

It's not that VFS-like, it's more a set of callback than actual methods. But yes, if you're actually doing work at a significant lower layer adding abstractions make sense. Note that seq_file.c while not beeing the VFS is also a generic library that you can use with any filesystem if you want to implement sequential synthetic files.

#97 lore raw

Re: reiser4 plugins

In reply to Alexander Zarochentsev
Quoted 1 line · Alexander Zarochentsev

On Sunday 26 June 2005 21:02, Christoph Hellwig wrote:

[...]

Quoted 4 lines
Quoted 4 lines

David and Hans, I've read through my backlog a lot now, and I must say it's pretty pointless - you're discussing lots of highlevel what if and don't actually care about something as boring as actual technical details.

Quoted 8 lines
Quoted 5 lines

Hans has lots of very skillfull technical people like zam and vs, and maybe he should give them some freedom to sort out technical issues for a basic reiser4 merge, and one that is done we can turn back to discussion of advanced features and their implementation, hopefully with a few more arguments on both sides and a real technical discussion.

Unfortunately, this is not only a technical discussion... it is about linux development model too.

Then better separate the two, keeping the technical discussion on LKML and taking the development model stuff elsewhere.

Quoted 1 line

Well, about the plugins.

First step: Axe them. They are (at most) /configuration options/, that will have to get fixed meanings, available to all ReiserFS filesystems purely for compatibility reasons. Sure, do get wild on new options in your own experimental trees, and migrate what survives into the standard version (and then it'll be ReiserFS 4.1, 4.2, ..., 4.25, ...).

Quoted 2 lines
                         We can clean reiser4<->VFS interface up by setting 

per-vfs-object inode/dentry/super ops instead using of our own dispatcher.

Sounds reasonable.

Quoted 2 lines

So different reiser4 inodes/files will have different i_ops/f_ops. That would be only visible-in-VFS part of reiser4 object plugins.

How would that work out from the userland (system call) perspective? How does that get handled from on-disk format?

Quoted 5 lines

Would the help to solve "reiser4 plugins" question? It is just as other FS do -- procfs has seq_file and sysconfig interfaces below the VFS and l-k people do not complain each day about layering violation ;-) Procfs is taken as an example because it deals with objects of different types, actually anybody may create own procfs objects more or less general way.

But procfs /is/ quite special, as it is supposed to be a window into the kernel, not real files. Some layering violation is unavoidable there.

Quoted 2 lines

I don't belive that you want to see all reiser4-specific things as item plugins, disk format plugins in the VFS.

Only what makes sense. Plus many of those will probably have to go. Decide on /one/ way of doing things, even if not perfect for all uses. Everything else is useless bloat.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#98 lore raw

Re: reiser4 plugins

Quoted 6 lines · Horst von Brand
Quoted 2 lines

I don't belive that you want to see all reiser4-specific things as item plugins, disk format plugins in the VFS.

Only what makes sense. Plus many of those will probably have to go. Decide on /one/ way of doing things, even if not perfect for all uses. Everything else is useless bloat.

It doesn't seem to be a problem as long as loadable plugins are GPL. How is it useless bloat? It doesn't seem much different from having loadable modules in the security system. Just don't enable Reiser4 in your kernel if you don't want to use it.

Sean

#99 lore raw

Re: reiser4 plugins

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 11 lines · Sean
Quoted 6 lines · Horst von Brand
Quoted 2 lines

I don't belive that you want to see all reiser4-specific things as item plugins, disk format plugins in the VFS.

Only what makes sense. Plus many of those will probably have to go. Decide on /one/ way of doing things, even if not perfect for all uses. Everything else is useless bloat.

It doesn't seem to be a problem as long as loadable plugins are GPL. How

"Loadable" how?

reiser4 plugins must be compiled in, either directly to the kernel, or part of reiser4.ko

The only thing "loadable" about them is that some can be turned on or off for individual files.

Quoted 3 lines

is it useless bloat? It doesn't seem much different from having loadable modules in the security system. Just don't enable Reiser4 in your kernel if you don't want to use it.

This is not getting us anywhere. It seems to be an issue of overall design and neatness at this point.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQsGkwXgHNmZLgCUhAQKUlw/6AxzaT+5R7lzhy4d/W55KY7m+8QJA90Ut 3m+E/hiSKykolERURKUoRPEK+jly8DUxLS+UpA5ZQwxzkR4YipMcoW5i3lHyJ9GB Hf9lBtGu0TwCF8W9Qh1Fk/7mwQcvyz6ATUYOYqvSA5Wxfr8BDFdcJUqTwMRCOxrA Fh5oOSBXEhqnSElfvurBkzMNurOmNnkpeZt8rM6qXqeYRRPrmIWmFGvkufoyb/q7 N2wdwtwoubWokl8FLWHQmH0f4pJqJUNXpDQvw67zKi24zOUq8ZZ7xH7EE8vNI9s9 ppVVsrLiAFE2/GeDRIdbQNSuClliq7XZV4AMHkorik7Qe0nIeR4hBr9hYI41l/Ie G3A9yP/69XsOodUWMZB/asxj3zMEhDynz26t98C2S858eRA7EJLcJ+IU1HnxHv7N KUcZLUyMP6DPDcm3r2FtjvwDwneQwk3zBqFE30DgAERZTgZ7RTglVuyBhETryk3m ZIf6bjuwnqn2/TFC7Yqwf9tzHshx91ywkfpOXwc/3IcrKKpfY74Mp4Cj/LkRemoz LnE6DdpjuNpMmwTwV0Vb5FKkfeO2nFkLZ96z8HphNmAAINh7gP3bpUMaODc8fLDG HtpzloYmBa4Yvn9JX+AwFuQStupPmScwa62Gcyv0Zm5EQ6fB1pqY1HdhpLHvXehT l0i+8OO74B0=
=NI4M
-----END PGP SIGNATURE-----

#100 lore raw

Re: reiser4 plugins

In reply to Jeff Garzik
Quoted 1 line · Jeff Garzik

We have to maintain said ugly code for decades.

No you don't, I do.

Quoted 5 lines
Quoted 4 lines

filesystem, but if so, it will have to do it much more slowly. Take the good ideas -- things like plugins -- and make them at least look like incremental updates to the current VFS, and make them available to all filesystems.

Quoted 8 lines

So, Reiser4 may eventually take over VFS and be the only Linux

This is how all Linux development is done.

The code evolves over time.

You have just described the path ReiserFS needs to take, in order to get this stuff into the kernel, in fact.

You missed his point. He is saying ext3 code should migrate towards becoming reiser4 plugins, and reiser4 should be merged now so that the migration can get started.

I don't really care whether ext3 uses our plugin model. (If it does, fine, please credit me for it though, and consider maybe a thank you. I am happy to thank the XFS team for the idea of delayed allocation....)

I am entitled to get some advantage from being the first on the block. Other fs maintainers don't want me to have that entitlement, so they add delays to slow us down. It is odd how Hellwig no longer describes himself as XFS associated. Did SGI and him

Quoted 1 line

There is no entitlement to be merged in the upstream kernel.

True, but what is interesting is that you don't think the faster filesystem should be merged, and the others should catch up to it later if they can, instead you think it should be delayed until all of its benefits can be transmitted to the other filesystems first so that everyone can be equal.

#101 lore raw

Re: reiser4 plugins

Hi Hans,

Quoted 1 line · Jeff Garzik
Quoted 1 line

We have to maintain said ugly code for decades.

Quoted 1 line · Hans Reiser

No you don't, I do.

Lots of people work on the kernel. If you wish to keep reiser4 maintenance to yourself, you probably need to keep it as a separate patch. Please consider submitting the non-controversial bits for merging first and then continue with the rest. It does not make a lot of sense reviewing the current patchkit as many parts of it won't be merged as is (e.g. the plugin stuff).

Pekka

#102 lore raw

Re: reiser4 plugins

In reply to Hans Reiser

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 20 lines · Hans Reiser
Quoted 1 line · Jeff Garzik

We have to maintain said ugly code for decades.

No you don't, I do.

Quoted 11 lines
Quoted 4 lines

filesystem, but if so, it will have to do it much more slowly. Take the good ideas -- things like plugins -- and make them at least look like incremental updates to the current VFS, and make them available to all filesystems.

So, Reiser4 may eventually take over VFS and be the only Linux

This is how all Linux development is done.

The code evolves over time.

Quoted 7 lines
Quoted 2 lines

You have just described the path ReiserFS needs to take, in order to get this stuff into the kernel, in fact.

You missed his point. He is saying ext3 code should migrate towards becoming reiser4 plugins, and reiser4 should be merged now so that the migration can get started.

Sort of.

I think ext3 would be nice as a reiser4 plugin. Not everyone will want to reformat at once, but as the reiser4 code matures and proves itself (even more than it already has), the ext3 people may find themselves wanting some of the more generic optimizations.

But, I don't think that will realistically happen at all.

Instead, what will probably happen is that once Reiser4 is in the mainstream kernel, it will become more popular and noticable. Other FSes will take note. ext3 people may decide they want file-as-directory, and vfat people may decide they want cryptocompress, and so on. In order to do this, they can work with Namesys to port whatever feature they need at the time to the standard VFS.

Eventually, with all the features ported, we end up with a situation where there may be no meaningful difference between a filesystem and a low-level reiser4 plugin.

At that point, we both win. ext3 will effectively be reiser4 plugins, and reiser4 will replace VFS, but incrementally and sanely, without breaking old filesystems or depriving them of features. This is indeed the way to evolve the code slowly over time.

This way also has the advantage of avoiding yet another, even more vicious flamefest if all the reiser4 functionality had to be ported to VFS all at once. Can any of you honestly say that you would be more willing to accept a brand-new, relatively unproven filesystem with tons of changes to the VFS than without?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrrDBHgHNmZLgCUhAQJN/A/6A4SqpsunKmj6NOKxqyMxkLVdMjxSuusa V9XZIL6K0dDsF1FmvxsSVnl3so8u93sRCkxrBs3WQs9JqsA3f8O5VUvsdfqas/Fb ZO+ht5n0r1BVfm1WJHHmR9xGm+jL2TMwVlVrcrrnBQLthq+9efggn5r9BXkOUPgz 57VMh6sjB9ioigmc2Wf6BsMlbBJpCz7EB91pcIrNzHdFOEWyuHm9mAP6ui2sppfK bGaDlZcU4BzNUM8beakMkFuy7JCTFQj/O7s3G42Tqpi/4Ol1Vk7F2tgPkpxB0lHm b/hCmCPbTHheRsDhT7gSHOzWrwA5pwKlT5nO4EWGq7DxZo2dgoYsa/2fP32uYPIX r3vUfygk7MR59DKMi4LLcDy2OzvrtbwTkzgOVrw82NCi5J5/pu/gIHH1aHzkXM6T YRd6G39+wmhN0n/MP3WAT7o3rbH4nuPiG8ZyZQyVTII9YRaOGwNLSjJKe1p9JFHY Y+6bqK3h7LQrX4MfTSIu8k3m0iIt806uHP3GlV+HHKx/N/4HSxt6ikikI3jURO4z XBr23/Uo3YNK+IuxQtPw2mKcdqqKhFq5OBmLkVFIzJyuh8KR7cfnFdl7luCTjFGL KjtgZ39SwtjOUWKWJslkcrNwESqplMTwAaB6GPHiJRKbilaCM9GXm/teI0ZRCl+o gBD0AdEBA9s=
=NJM0
-----END PGP SIGNATURE-----

#103 lore raw

Re: reiser4 plugins

Quoted 2 lines · David Masover
Quoted 1 line · Hans Reiser

Jeff Garzik wrote:

[...]

Quoted 7 lines
Quoted 3 lines

You missed his point. He is saying ext3 code should migrate towards becoming reiser4 plugins, and reiser4 should be merged now so that the migration can get started.

Sort of.

I think ext3 would be nice as a reiser4 plugin.

What for? It works just fine as it stands, AFAICS.

Quoted 3 lines

Not everyone will want
to reformat at once, but as the reiser4 code matures and proves itself (even more than it already has),

I for one have seen mainly people with wild claims that it will make their machines much faster, and coming back later asking how they can recover their thrashed partitions...

Quoted 2 lines

the ext3 people may find themselves
wanting some of the more generic optimizations.

They'll filch them in due time, don't worry.

Quoted 6 lines

But, I don't think that will realistically happen at all.

Instead, what will probably happen is that once Reiser4 is in the mainstream kernel, it will become more popular and noticable. Other FSes will take note. ext3 people may decide they want file-as-directory,

That idea is even much older than Linux itself, and no other (Unix) filesystem has implemented it. Ever. Wonder why...

Quoted 1 line

and vfat people may decide they want cryptocompress,

I'm sure they don't, because it is mostly for Windows and assorted devices (pendrives, digital cameras, ...) compatibility.

Quoted 2 lines

and so on. In order to do this, they can work with Namesys to port whatever feature they need at the time to the standard VFS.

I'm sure they are grateful for the offer.

Quoted 3 lines

Eventually, with all the features ported, we end up with a situation where there may be no meaningful difference between a filesystem and a low-level reiser4 plugin.

Could very well take decades, if ever.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#104 lore raw

Re: reiser4 plugins

Quoted 8 lines
Quoted 4 lines

Not everyone will want
to reformat at once, but as the reiser4 code matures and proves itself (even more than it already has),

I for one have seen mainly people with wild claims that it will make their machines much faster, and coming back later asking how they can recover their thrashed partitions...

Then please show us some Links/Message-IDs to such postings. I'd like to read them.

From my POV:
I've been using Reiser4 for almost everything (Rootfs / External Harddrives) for about ~8 Months without any data loss..

Powerloss, unpluging the Disk while writing, full filesystem, heavy use : No problems with reiser4.. It *is* stable.

-- Adrian

#105 lore raw

Re: reiser4 plugins

Quoted 10 lines
Quoted 7 lines
Quoted 3 lines

Not everyone will want
to reformat at once, but as the reiser4 code matures and proves itself (even more than it already has),

I for one have seen mainly people with wild claims that it will make their machines much faster, and coming back later asking how they can recover their thrashed partitions...

Then please show us some Links/Message-IDs to such postings. I'd like to read them.

Here you are....

The following happened to me with reiserfs as it was shipped with suse 9.1:

------------------------------------------------------------------
dreher@euler03:~/mytex/konstanz/wohnung> ls
auto makler2.aux makler2.log makler2.tex makler.aux makler.log
swk.eps unilogo.eps
briefkpf.tex makler2.dvi makler2.ps makler3.tex makler.dvi makler.tex
unikopf.tex
dreher@euler03:~/mytex/konstanz/wohnung> rm *.aux *.log
rm: cannot remove `makler2.log': No such file or directory dreher@euler03:~/mytex/konstanz/wohnung> ls
auto briefkpf.tex makler2.dvi makler2.ps makler2.tex makler3.tex makler.dvi makler.tex swk.eps unikopf.tex unilogo.eps dreher@euler03:~/mytex/konstanz/wohnung> uname -a
Linux euler03 2.6.5-7.108-smp #1 SMP Wed Aug 25 13:34:40 UTC 2004 i686 i686 i386 GNU/Linux
dreher@euler03:~/mytex/konstanz/wohnung> date
Tue Sep 21 13:15:45 CEST 2004
----------------------------------------------------------

Note the line "rm: cannot remove `makler2.log': No such file or directory"

There was no data loss, but such a bug should not happen. I never had similar experiences with ext3.

Unfortunately, I cannot reproduce this behavior.

Quoted 2 lines

Powerloss, unpluging the Disk while writing, full filesystem, heavy use : No problems with reiser4.. It *is* stable.

My impression: reiser3 is not 100% stable, but quite stable, written by someone who asks for "review by benchmark".

Michael

#106 lore raw

Re: reiser4 plugins

I think we are talking about reiser4, not reiser3..

Michele

Quoted 54 lines · Michael Dreher
Quoted 10 lines
Quoted 7 lines
Quoted 3 lines

Not everyone will want
to reformat at once, but as the reiser4 code matures and proves itself (even more than it already has),

I for one have seen mainly people with wild claims that it will make their machines much faster, and coming back later asking how they can recover their thrashed partitions...

Then please show us some Links/Message-IDs to such postings. I'd like to read them.

Here you are....

The following happened to me with reiserfs as it was shipped with suse 9.1:

------------------------------------------------------------------
dreher@euler03:~/mytex/konstanz/wohnung> ls
auto makler2.aux makler2.log makler2.tex makler.aux makler.log
swk.eps unilogo.eps
briefkpf.tex makler2.dvi makler2.ps makler3.tex makler.dvi makler.tex
unikopf.tex
dreher@euler03:~/mytex/konstanz/wohnung> rm *.aux *.log
rm: cannot remove `makler2.log': No such file or directory dreher@euler03:~/mytex/konstanz/wohnung> ls
auto briefkpf.tex makler2.dvi makler2.ps makler2.tex makler3.tex makler.dvi makler.tex swk.eps unikopf.tex unilogo.eps dreher@euler03:~/mytex/konstanz/wohnung> uname -a
Linux euler03 2.6.5-7.108-smp #1 SMP Wed Aug 25 13:34:40 UTC 2004 i686 i686 i386 GNU/Linux
dreher@euler03:~/mytex/konstanz/wohnung> date
Tue Sep 21 13:15:45 CEST 2004
----------------------------------------------------------

Note the line "rm: cannot remove `makler2.log': No such file or directory"

There was no data loss, but such a bug should not happen. I never had similar experiences with ext3.

Unfortunately, I cannot reproduce this behavior.

Quoted 2 lines

Powerloss, unpluging the Disk while writing, full filesystem, heavy use : No problems with reiser4.. It *is* stable.

My impression: reiser3 is not 100% stable, but quite stable, written by someone who asks for "review by benchmark".

Michael
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#107 lore raw

Re: reiser4 plugins

In reply to Michael Dreher
Quoted 2 lines · Michael Dreher

My impression: reiser3 is not 100% stable, but quite stable, written by someone who asks for "review by benchmark".

Review by uniprocessor benchmark perhaps. However Reiser4 is new code. The original extfs on Linux was not very good either while ext2 was excellent. It seems inappropriate to technically review one fs based on the other. Looking at the authors maintenance record I think is important but every single star Linux kernel contributors first major contribution was generally not very good.

Alan

#108 lore raw

Re: reiser4 plugins

In reply to Michael Dreher

One, you were using V3 not V4.

Two, this bug you mention is probably not an fs bug. rm first creates a list of names, and then removes them.

reiser@linux:~/scratch> touch fufu reiser@linux:~/scratch> touch fifu reiser@linux:~/scratch> rm *fu fi*
rm: cannot remove `fifu': No such file or directory

Your file either somehow got removed before rm got to it, or rm somehow got to it twice. Vitaly, can you look at the error handling by rm and see if it can get to things twice when it hits an error or if you can otherwise figure this out? If I remember right, I have hit this myself for non-reiserfs filesystems, and I never investigated it.

Quoted 60 lines · Michael Dreher
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Not everyone will want
to reformat at once, but as the reiser4 code matures and proves itself (even more than it already has),

Quoted 9 lines
Quoted 5 lines

I for one have seen mainly people with wild claims that it will make their machines much faster, and coming back later asking how they can recover their thrashed partitions...

Then please show us some Links/Message-IDs to such postings. I'd like to read them.

Here you are....

The following happened to me with reiserfs as it was shipped with suse 9.1:

------------------------------------------------------------------
dreher@euler03:~/mytex/konstanz/wohnung> ls
auto makler2.aux makler2.log makler2.tex makler.aux makler.log
swk.eps unilogo.eps
briefkpf.tex makler2.dvi makler2.ps makler3.tex makler.dvi makler.tex
unikopf.tex
dreher@euler03:~/mytex/konstanz/wohnung> rm *.aux *.log
rm: cannot remove `makler2.log': No such file or directory dreher@euler03:~/mytex/konstanz/wohnung> ls
auto briefkpf.tex makler2.dvi makler2.ps makler2.tex makler3.tex makler.dvi makler.tex swk.eps unikopf.tex unilogo.eps dreher@euler03:~/mytex/konstanz/wohnung> uname -a
Linux euler03 2.6.5-7.108-smp #1 SMP Wed Aug 25 13:34:40 UTC 2004 i686 i686 i386 GNU/Linux
dreher@euler03:~/mytex/konstanz/wohnung> date
Tue Sep 21 13:15:45 CEST 2004
----------------------------------------------------------

Note the line "rm: cannot remove `makler2.log': No such file or directory"

There was no data loss, but such a bug should not happen. I never had similar experiences with ext3.

Unfortunately, I cannot reproduce this behavior.

Quoted 4 lines

Powerloss, unpluging the Disk while writing, full filesystem, heavy use : No problems with reiser4.. It *is* stable.

My impression: reiser3 is not 100% stable, but quite stable, written by someone who asks for "review by benchmark".

Michael

#109 lore raw

Re: reiser4 plugins

Quoted 4 lines · In article <[email protected]> you wrote:

reiser@linux:~/scratch> touch fufu reiser@linux:~/scratch> touch fifu reiser@linux:~/scratch> rm *fu fi*
rm: cannot remove `fifu': No such file or directory

# echo rm *fu fi*
rm fifu fufu fifu

shell magic :)

Bernd

#110 lore raw

Re: reiser4 plugins

Quoted 8 lines · Bernd Eckenfels
Quoted 4 lines

reiser@linux:~/scratch> touch fufu reiser@linux:~/scratch> touch fifu reiser@linux:~/scratch> rm *fu fi*
rm: cannot remove `fifu': No such file or directory

# echo rm *fu fi*
rm fifu fufu fifu
shell magic :)

may i add that you guys are a bunch of stoners :)

#111 lore raw

Re: reiser4 plugins

In reply to Hans Reiser
Quoted 4 lines · Hans Reiser

One, you were using V3 not V4.

Two, this bug you mention is probably not an fs bug. rm first creates a list of names, and then removes them.

shell creates a list of names, rm simply handles the given set of names and tries to remove a file twice if specified twice.

the problem could be in the shell, or what is more likely in patterns that overlap each other, like in the example of 'rm *fu fi*'. we should probably investigate a reproducable example.

Quoted 8 lines

reiser@linux:~/scratch> touch fufu reiser@linux:~/scratch> touch fifu reiser@linux:~/scratch> rm *fu fi*
rm: cannot remove `fifu': No such file or directory

Your file either somehow got removed before rm got to it, or rm somehow got to it twice. Vitaly, can you look at the error handling by rm and see if it can get to things twice when it hits an error or if you can

once per every given name.

Quoted 68 lines

otherwise figure this out? If I remember right, I have hit this myself for non-reiserfs filesystems, and I never investigated it.

Quoted 60 lines · Michael Dreher
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines

Not everyone will want
to reformat at once, but as the reiser4 code matures and proves itself (even more than it already has),

Quoted 9 lines
Quoted 5 lines

I for one have seen mainly people with wild claims that it will make their machines much faster, and coming back later asking how they can recover their thrashed partitions...

Then please show us some Links/Message-IDs to such postings. I'd like to read them.

Here you are....

The following happened to me with reiserfs as it was shipped with suse 9.1:

------------------------------------------------------------------
dreher@euler03:~/mytex/konstanz/wohnung> ls
auto makler2.aux makler2.log makler2.tex makler.aux makler.log
swk.eps unilogo.eps
briefkpf.tex makler2.dvi makler2.ps makler3.tex makler.dvi makler.tex
unikopf.tex
dreher@euler03:~/mytex/konstanz/wohnung> rm *.aux *.log
rm: cannot remove `makler2.log': No such file or directory dreher@euler03:~/mytex/konstanz/wohnung> ls
auto briefkpf.tex makler2.dvi makler2.ps makler2.tex makler3.tex makler.dvi makler.tex swk.eps unikopf.tex unilogo.eps dreher@euler03:~/mytex/konstanz/wohnung> uname -a
Linux euler03 2.6.5-7.108-smp #1 SMP Wed Aug 25 13:34:40 UTC 2004 i686 i686 i386 GNU/Linux
dreher@euler03:~/mytex/konstanz/wohnung> date
Tue Sep 21 13:15:45 CEST 2004
----------------------------------------------------------

Note the line "rm: cannot remove `makler2.log': No such file or directory"

There was no data loss, but such a bug should not happen. I never had similar experiences with ext3.

Unfortunately, I cannot reproduce this behavior.

Quoted 4 lines

Powerloss, unpluging the Disk while writing, full filesystem, heavy use : No problems with reiser4.. It *is* stable.

My impression: reiser3 is not 100% stable, but quite stable, written by someone who asks for "review by benchmark".

Michael

Signature · 2 lines
Thanks,
Vitaly Fertman
#112 lore raw

Re: reiser4 plugins

In reply to Adrian Ulrich
Quoted 6 lines · Adrian Ulrich

From my POV:
I've been using Reiser4 for almost everything (Rootfs / External Harddrives) for about ~8 Months without any data loss..

Powerloss, unpluging the Disk while writing, full filesystem, heavy use : No problems with reiser4.. It *is* stable.

*From users who use it* I have heard nothing but love for reiser4. It's amazing how quickly people seem to be dismissive about what reiser4 has to offer when they more than likely haven't taken it for a spin at all. All I hear about is 'we can't let something ugly go into the stable kernel' then in the same day I looked into some of the config options...

CONFIG_WDC_ALI15X3:
*snip*
This allows for UltraDMA support for WDC drives that ignore CRC checking. You are a fool for enabling this option, but there have been requests.
*snip*

How many have requested that reiser4 make it into the kernel? I'd imagine many more then requested this IDE feature. And it's an *option*. Please work something out on this.

avuton

Signature · 1 line
  Anyone who quotes me in their sig is an idiot. -- Rusty Russell.
#113 lore raw

Re: reiser4 plugins

Quoted 24 lines · Avuton Olrich
Quoted 6 lines · Adrian Ulrich

From my POV:
I've been using Reiser4 for almost everything (Rootfs / External Harddrives) for about ~8 Months without any data loss..

Powerloss, unpluging the Disk while writing, full filesystem, heavy use : No problems with reiser4.. It *is* stable.

*From users who use it* I have heard nothing but love for reiser4. It's amazing how quickly people seem to be dismissive about what reiser4 has to offer when they more than likely haven't taken it for a spin at all. All I hear about is 'we can't let something ugly go into the stable kernel' then in the same day I looked into some of the config options...

CONFIG_WDC_ALI15X3:
*snip*
This allows for UltraDMA support for WDC drives that ignore CRC checking. You are a fool for enabling this option, but there have been requests.
*snip*

Quoted 6 lines

How many have requested that reiser4 make it into the kernel? I'd imagine many more then requested this IDE feature. And it's an *option*. Please work something out on this.

avuton

As a user who uses ReiserFS v3, I have nothing but love for ReiserFS.

The speed and reliability are great right now, I'm really looking forward to using ReiserFS v4.

--Dan

#114 lore raw

Re: reiser4 plugins

In reply to Avuton Olrich
Quoted 15 lines · Avuton Olrich
Quoted 6 lines · Adrian Ulrich

From my POV:
I've been using Reiser4 for almost everything (Rootfs / External Harddrives) for about ~8 Months without any data loss..

Powerloss, unpluging the Disk while writing, full filesystem, heavy use : No problems with reiser4.. It *is* stable.

*From users who use it* I have heard nothing but love for reiser4. It's amazing how quickly people seem to be dismissive about what reiser4 has to offer when they more than likely haven't taken it for a spin at all. All I hear about is 'we can't let something ugly go into the stable kernel' then in the same day I looked into some of the config options...

Quoted 6 lines

CONFIG_WDC_ALI15X3:
*snip*
This allows for UltraDMA support for WDC drives that ignore CRC checking. You are a fool for enabling this option, but there have been requests.
*snip*

That's hardly a good example, that config option, while obviously questionable, only added 2 #ifdef blocks and affects 1 function that's 20 lines long.

Quoted 6 lines

How many have requested that reiser4 make it into the kernel? I'd imagine many more then requested this IDE feature. And it's an *option*. Please work something out on this.

avuton

Jim.

#115 lore raw

Re: reiser4 plugins

In reply to Horst von Brand

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 15 lines · Horst von Brand
Quoted 3 lines · David Masover
Quoted 1 line · Hans Reiser

Jeff Garzik wrote:

[...]

Quoted 3 lines
Quoted 3 lines

You missed his point. He is saying ext3 code should migrate towards becoming reiser4 plugins, and reiser4 should be merged now so that the migration can get started.

Quoted 6 lines
Quoted 3 lines

Sort of.

I think ext3 would be nice as a reiser4 plugin.

What for? It works just fine as it stands, AFAICS.

So does DOS. Do you use DOS? I don't even use DOS to run DOS programs.

"Ain't broke" is the battle cry of stagnation.

But, there are some things Reiser does better and faster than ext3, even if you don't count file-as-directory and other toys. There's nothing ext3 does better than Reiser, unless you count the compatibility with random bootloaders and low-level tools.

Quoted 8 lines
Quoted 3 lines

Not everyone will want
to reformat at once, but as the reiser4 code matures and proves itself (even more than it already has),

I for one have seen mainly people with wild claims that it will make their machines much faster, and coming back later asking how they can recover their thrashed partitions...

You know how many I've had thrashed on Reiser4? Two. The first one was with a VERY early alpha/beta, and the second one was when I dropped a laptop and the disk failed.

And it does make certain things faster. For one thing, "emerge sync" on Gentoo is twice to four times as fast, and /usr/portage is 75% as big, as on ReiserFS (3).

Quoted 5 lines
Quoted 2 lines

the ext3 people may find themselves
wanting some of the more generic optimizations.

They'll filch them in due time, don't worry.

Duplication of effort. With plugins, we can optimize the upper layers of ALL filesystems, regardless of the lower layers, in such a way that it is optional. I'm sure it's far easier to write a Reiser storage plugin than a brand new FS.

Eventually, once competition is only based on storage format, we could end up with just one format. Just one filesystem! (except for fat/ntfs/iso/udf/network...) And in the open source world, sometimes a single product is a good thing.

Quoted 12 lines
Quoted 6 lines

But, I don't think that will realistically happen at all.

Instead, what will probably happen is that once Reiser4 is in the mainstream kernel, it will become more popular and noticable. Other FSes will take note. ext3 people may decide they want file-as-directory,

That idea is even much older than Linux itself, and no other (Unix) filesystem has implemented it. Ever. Wonder why...

Quoted 5 lines
Quoted 1 line

and vfat people may decide they want cryptocompress,

I'm sure they don't, because it is mostly for Windows and assorted devices (pendrives, digital cameras, ...) compatibility.

I, for one, would like to use a pendrive and have certain files be encrypted transparently, only for use on Linux, but others be ready to transfer to a Windows box.

Quoted 6 lines
Quoted 3 lines

Eventually, with all the features ported, we end up with a situation where there may be no meaningful difference between a filesystem and a low-level reiser4 plugin.

Could very well take decades, if ever.

How would you do it, in a way that doesn't take decades? -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrsx6HgHNmZLgCUhAQKXYw//Z9bSh74AWr2o1n+EM0nCHUNV78xgeuey ZHtzUT8rCv2KQ+2fMe5EnRUaRYTKvFnnGccH4vu/OAArqKqt/RgP3NP8UZALsZbB MMoEHZSX/E0BFlKKiBjPgvwfnnY9ZYF0GPkU5L97dj1K0dEQMndOZoYDV07H4TnN /1VkytnsXuhm5nqRhPd89rDbvQtXpzHiDjVNPfT+J6M6JFw8jfYQZ0Fo1T9dVKMg qibVneJj+onHVBmBBqGTZ0Ane5VmG0h0a2ZwsslQPkf03DAgprliykr40NCECdli OdaS73qYPlYRRl1nuw84g2KOLXbTnSGmW1t4qt/tyI6t3TOEn9FqY7YzwvbWIVLP GMkJG1htAQefGNcEXx+15xAHJi6/0DiSoJu+P+ie+yG8pkG943936AxEgs89pTpC z8i/GV9Uq+S6BgA+RJuxpk6U8rQ0YHMhAiK83p+s7vdUsGIKomUmSbn7a6DC7cZt aGkqxYyaoV2MlHUMUebSv5F82JUgx6rPuc8SQW1wvVIVdNA7QhlYYsPa5vFFj34C scxN9vNGyxWGu60yKx7LSYRkB9/prJytKvpezVKRkn8pnKnl4AHodKioSPxt1iHC BvNsLL8Kn8FBe/HG98d1v8vwTqe0Y8KgBMRe/J73u/OzU82lh2V3YbXNaW+DLVGz MOaHYEzfoNg=
=yFsv
-----END PGP SIGNATURE-----

#116 lore raw

Re: reiser4 plugins

Quoted 3 lines · David Masover
Quoted 1 line

What for? It works just fine as it stands, AFAICS.

So does DOS. Do you use DOS? I don't even use DOS to run DOS programs.

False argument. So does the pen, so do hinges on doors. Do you still have hinges on your doors - probably.

Quoted 1 line

"Ain't broke" is the battle cry of stagnation.

Its also the battle cry of everyone over the age of 20 who also has a real job to do 8)

Quoted 4 lines

But, there are some things Reiser does better and faster than ext3, even if you don't count file-as-directory and other toys. There's nothing ext3 does better than Reiser, unless you count the compatibility with random bootloaders and low-level tools.

Certainly compared with reiser3 you've missed a few out including resilience to disk errors (nearly nil on reiser3), and SMP scaling.

Quoted 3 lines

You know how many I've had thrashed on Reiser4? Two. The first one was with a VERY early alpha/beta, and the second one was when I dropped a laptop and the disk failed.

Entirely or bad blocks ? The latter should have a minimal cost on a well designed fs.

Quoted 2 lines

Duplication of effort. With plugins, we can optimize the upper layers of ALL filesystems, regardless of the lower layers, in such a way that

In which case the features belong in the VFS as all those with experience and kernel contributions have been arguing.

#117 lore raw

Re: reiser4 plugins

Quoted 1 line · Alan Cox

SMP scaling.

Reiser4 should do much better at this, as it was designed for it. I wish we had a nice hunking multiprocessor to verify that and work through the inevitable unintended sources of bottlenecks though.

Quoted 13 lines
Quoted 7 lines

You know how many I've had thrashed on Reiser4? Two. The first one was with a VERY early alpha/beta, and the second one was when I dropped a laptop and the disk failed.

Entirely or bad blocks ? The latter should have a minimal cost on a well designed fs.

Quoted 9 lines
Quoted 4 lines

Duplication of effort. With plugins, we can optimize the upper layers of ALL filesystems, regardless of the lower layers, in such a way that

In which case the features belong in the VFS as all those with experience and kernel contributions have been arguing.

So you fundamentally reject the prototype it in one fs and then abstract it to others development model?

#118 lore raw

Re: reiser4 plugins

Quoted 9 lines · Hans Reiser
Quoted 5 lines · Alan Cox
Quoted 2 lines

Duplication of effort. With plugins, we can optimize the upper layers of ALL filesystems, regardless of the lower layers, in such a way that

In which case the features belong in the VFS as all those with experience and kernel contributions have been arguing.

So you fundamentally reject the prototype it in one fs and then abstract it to others development model?

For similar reasons, we don't let hardware vendors implement software RAID inside the hardware driver. It's not appropriate at that level.

Jeff

#119 lore raw

Re: reiser4 plugins

In reply to Hans Reiser
Quoted 4 lines · Hans Reiser

So you fundamentally reject the prototype it in one fs and then abstract it to others development model?

Hans,
after many years here now, one would have thought you would have "got" this part of Linux: kernel development & code that gets into the kernel only does so by getting past the benevolent dictators. instead, it seems that every time there is ReiserFS to be merged (and we can go back in history a number of years here..), it always seems to come as a great shock that your code won't be merged 'as-is' without peer review & comment.

don't feel that you're being singled out here. you aren't. there isn't any anti-Hans-and-his-filesystem conspiracy here.
there are plenty of examples on where this has happened in Linux previously in other parts of the tree.
EVMS is a great example of similar things - a proposal to include kernel code to do various volume-mgmt functions - which was basically accomplishing the same goal as that of LVM/LVM2 and MD drivers (& DM framework). the EVMS team are a great act to follow - see http://lwn.net/Articles/14714/ - they showed high levels of professional conduct and made what was essentially a 'hard' but 'correct' decision in reworking EVMS to use the same DM infrastructure as LVM2. there are countless other examples at various times - various 'competing' IPv6 projects, IPSec, various "hardware" (software) RAID controllers, various IP offload schemes et al.

why does Reiserfs have to be any different?

you know that VFS is the mechanism in Linux. you know (i hope..) how it works. it isn't so hard to see how many of the Reiser4 "plug-ins" could be tied into VFS calls. OR, if they cannot TODAY, propose how VFS _COULD_ be made to do this.

the key here is trust. and trust is a two-way street.

the irony of this whole thread is that history is repeating itself. see http://www.ussg.iu.edu/hypermail/linux/kernel/0112.1/0519.html
kernel developers pushed back on you 3 years ago - in 2001 - what has really changed?

*an observation*

cheers,

lincoln.

#120 lore raw

Re: reiser4 plugins

Quoted 1 line · Lincoln Dale

the EVMS team are a great act to follow - see http://lwn.net/Articles/14714/ - they showed high levels of professional conduct and made what was essentially a 'hard' but 'correct' decision in reworking EVMS to use the same DM infrastructure as LVM2.

I just want to heartily second this. The EVMS team made the hard but ultimately best choice, and everybody benefited. They deserve all the kudos they get.

Jeff

#121 lore raw

Re: reiser4 plugins

In reply to Lincoln Dale
Quoted 5 lines · Lincoln Dale

the irony of this whole thread is that history is repeating itself. see http://www.ussg.iu.edu/hypermail/linux/kernel/0112.1/0519.html
kernel developers pushed back on you 3 years ago - in 2001 - what has really changed?

It is exactly the same, but rather than dwell on that, I'll just remind that I have sent out two technical emails which talk only about the issue of plugins and pluginids, and whether plugins are classes rather than just instances, and whether the classes really would benefit from being instantiated into VFS at the cost of keeping the same info in two places, and I got no answer on them. Zam pointed out that our plugins do more than just VFS operations, and got no response on that or his other points.

Regarding trust, Christophe comes out the gate using the words "useless abstraction layer" that happens to be a core feature of our design, demanding we cut it out, and not really understanding it or recognizing any functionality it provides, and you can't really expect me to respect this, can you?

Now, if his target is reduced to whether we can eliminate a function indirection, and whether we can review the code together and see if it is easy to extend plugins and pluginids to other filesystems by finding places to make it more generic and accepting of per filesystem plugins, especially if it is not tied to going into 2.6.13, well, that is the conversation I would have liked to have had.

#122 lore raw

Re: reiser4 plugins

Quoted 19 lines · Hans Reiser
Quoted 5 lines · Lincoln Dale

the irony of this whole thread is that history is repeating itself. see http://www.ussg.iu.edu/hypermail/linux/kernel/0112.1/0519.html
kernel developers pushed back on you 3 years ago - in 2001 - what has
really changed?

It is exactly the same, but rather than dwell on that, I'll just remind that I have sent out two technical emails which talk only about the issue of plugins and pluginids, and whether plugins are classes rather than just instances, and whether the classes really would benefit from being instantiated into VFS at the cost of keeping the same info in two places, and I got no answer on them. Zam pointed out that our plugins do more than just VFS operations, and got no response on that or his other points.

Quoted 7 lines

Regarding trust, Christophe comes out the gate using the words "useless abstraction layer" that happens to be a core feature of our design, demanding we cut it out, and not really understanding it or recognizing any functionality it provides, and you can't really expect me to respect this, can you?

heh. one example i didn't mention in the myriad of EVMS, IPv6, IPSec et al was iSCSI.
Christoph was (rightly so) very strong in pushing back on things that needed to be fixed/changed/addressed in linux-iscsi, a project near & dear to much of the paid work i do.

while i'm sure Christoph could probably be more tactful at times, his views on technical matters in the kernel are respected.
the key here is to not take it personally - but instead just understand that this is the framework that you have to work in to get it into the mainline kernel.

if you don't want to go down that path, you're free to do so. its open source, you can keep your own linux-kernel fork.
but if you want to get your code into mainline, i don't think its so much a case of l-k folks telling you how to make your code work under VFS. the onus is on you to say WHY your code and plugins could never be put into VFS.

if anything, you should be thankful that Al hasn't yet commented. :)

Quoted 9 lines

Now, if his target is reduced to whether we can eliminate a function indirection, and whether we can review the code together and see if it is easy to extend plugins and pluginids to other filesystems by finding places to make it more generic and accepting of per filesystem plugins, especially if it is not tied to going into 2.6.13, well, that is the conversation I would have liked to have had.

fantastic - some common ground.
any reason WHY there has to be an abstraction of 'pluginid' when in theory VFS operations can already provide the necessary abstraction on a per-object basis?

Nikita basically said as much in Message-ID: <[email protected]> earlier in this thread:
"But it is not so. There _are_ plugins-in-the-VFS. VFS operates on
opaque
objects (inodes, dentries, file system types) through interfaces:
{inode,address_space,dentry,sb,etc.}_operations. Every file system
back-end if free to implement whatever number of these interfaces. And
the do this already: check the sources; even ext2 does this: e.g., ext2_fast_symlink_inode_operations and ext2_symlink_inode_operations.

This is exactly what upper level reiser4 plugins are for.

I guess that one of Christoph Hellwig's complaints is that reiser4 introduces another layer of abstraction to implement something that already exists."

i never saw any reason why there can't be specific VFS operations put together for specific inodes if the policy for that inode dictates any special operation (e.g. compression).

cheers,

lincoln.

#123 lore raw

Re: reiser4 plugins

Quoted 10 lines · Lincoln Dale
Quoted 10 lines

Now, if his target is reduced to whether we can eliminate a function indirection, and whether we can review the code together and see if it is easy to extend plugins and pluginids to other filesystems by finding places to make it more generic and accepting of per filesystem plugins, especially if it is not tied to going into 2.6.13, well, that is the conversation I would have liked to have had.

Quoted 4 lines

fantastic - some common ground.
any reason WHY there has to be an abstraction of 'pluginid' when in theory VFS operations can already provide the necessary abstraction on a per-object basis?

VFS supplies instances, plugins are classes. If a language can instantiate an object, that does not eliminate the value of being able to create classes.

Does it make sense to you now?

#124 lore raw

Re: reiser4 plugins

Quoted 13 lines · Hans Reiser
Quoted 7 lines
Quoted 7 lines

ow, if his target is reduced to whether we can eliminate a function indirection, and whether we can review the code together and see if it is easy to extend plugins and pluginids to other filesystems by finding places to make it more generic and accepting of per filesystem plugins, especially if it is not tied to going into 2.6.13, well, that is the conversation I would have liked to have had.

Quoted 6 lines

fantastic - some common ground.
any reason WHY there has to be an abstraction of 'pluginid' when in theory VFS operations can already provide the necessary abstraction on a per-object basis?

Quoted 7 lines

VFS supplies instances, plugins are classes. If a language can instantiate an object, that does not eliminate the value of being able to create classes.

Does it make sense to you now?

you've lost me . . .

regardless, it isn't /me/ that you need to convince. how about a posting to l-k on "why Reiser4 cannot use VFS infrastructure for [crypto,compression,blahblah] plugins" - ideally, for each plugin.

cheers,

lincoln.

#125 lore raw

Re: reiser4 plugins

In reply to Hans Reiser
Quoted 8 lines · Hans Reiser
Quoted 6 lines · Lincoln Dale
Quoted 6 lines

Now, if his target is reduced to whether we can eliminate a function indirection, and whether we can review the code together and see if it is easy to extend plugins and pluginids to other filesystems by finding places to make it more generic and accepting of per filesystem plugins, especially if it is not tied to going into 2.6.13, well, that is the conversation I would have liked to have had.

Quoted 8 lines
Quoted 4 lines

fantastic - some common ground.
any reason WHY there has to be an abstraction of 'pluginid' when in theory VFS operations can already provide the necessary abstraction on a per-object basis?

VFS supplies instances, plugins are classes. If a language can instantiate an object, that does not eliminate the value of being able to create classes.

In OOP speak, VFS is an abstract class, each individual filesystem derives from this class giving a concrete class, a specific on-disk (or whereever) filesystem is an object of its (concrete) class. The rest of the kernel (as a client) doesn't care for the concrete classes, it speaks only (or mostly) in terms of the abstract class (VFS). And concrete filesystems in turn use the generic block layer.

Quoted 1 line

Does it make sense to you now?

No. Sounds jumbled up and backwards. And I don't see how "languages" could even enter the picture here.

Again: Is there any (sane) way the /existing/ VFS can be used to express what you want? What are the /minimal/ changes to VFS so each extension can be catered for in an uniform way across filesystems (anything else destroys the core idea of having a VFS in the first place!)? What are the required changes in the clients of VFS (i.e., changes to the core kernel)? What would be the impact on existing filesystems that /don't/ use new functionality (this is a lot harder, because it impacts many independent pieces of code)? What would be required for an existing filesystem to incorporate it?

When all this is answered, go over the implementation details. But that is far off still.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#126 lore raw

Re: reiser4 plugins

Quoted 13 lines · Horst von Brand
Quoted 4 lines · Hans Reiser

VFS supplies instances, plugins are classes. If a language can instantiate an object, that does not eliminate the value of being able to create classes.

In OOP speak, VFS is an abstract class, each individual filesystem derives from this class giving a concrete class, a specific on-disk (or whereever) filesystem is an object of its (concrete) class. The rest of the kernel (as a client) doesn't care for the concrete classes, it speaks only (or mostly) in terms of the abstract class (VFS). And concrete filesystems in turn use the generic block layer.

Quoted 4 lines
Quoted 1 line

Does it make sense to you now?

No. Sounds jumbled up and backwards. And I don't see how "languages" could even enter the picture here.

There seems to be a bit of confusion by the OO speak (I was initially confused as well -- guess I've spend too much time in broken OO languages recently). *grin* Thinking of interfaces, classes, and objects as a bunch of 1:M relationships make it much clearer.

If you start at the bottom, an object (instantiation of a class) has the per-object stuff (usually data). A class (instantiation of a meta-class or interface) has the shared object stuff (methods/functions that work on the data and static/shared data). An interface has the shared class stuff (info on what the methods/functions and static/shared data are).

A class is a realization of an interface. An object is a realization of a class. The standard C thing is to have only objects be dynamic at runtime. A slightly less common (but very flexible) thing to do is to make the classes dynamic at runtime. This introduces the following C -> OO talk:

-Instances of structures full of data are objects.
-Instances of structure full of function pointers are classes. -Instances of structure that are a big mix of both are the crossbred offsprings of programmers committing OO maintenance suicide by violating all the layering (or a savy programmers cleverly saving of one level of indirection by caching an object's class' function pointers in the object -- depends on your point of view *grin*).

In the Reiser4 FS I believe it goes like this (apologies Hans if I'm wrong). The objects are files, directories, etc. The classes are the plugins. Associated with each object is a number (plugin id/class pointer) identifying the set of functions (class/plugin) that provides the functions/methods to manipulate that object (ids are required because the objects [files, directories, etc.] live on the device while the plugins live in memory).

The good thing is that it is very easy to provide new plugins (instances of the interface to manipulate the objects). To create a new new plugin (a new class) you just fill in a bunch of function pointer fields in a structure (possibly mixing and matching existing functions) and give it a plugin id. To use it, you just link appropriate objects (files for file plugins, directories for directory plugins, etc.) back to it via the plugin id.

The VFS also has a bunch of OO stuff. Looking at "include/linux/fs.h", the inode structure is mostly data with a few pointers to structures full of functions pointers. Instances of it are most definitely objects. Instances of the associated structures full of functions pointers (inode_operations, file_operations, etc.) are most definitely classes (they give an implement of their respective interfaces for that inode). Likewise, instances of super_block are objects, while instances of super_operations (and friends) are classes. In short, in Reiser4 FS talk, much of the VFS is done with plugins (a plugin being an instance of one of the *_operations structures).

The current merge issue seems to be that apparently (I haven't actually looked myself) lots of the VFS plugins that Reiser4 exports are just thunks. This might imply that there are Reiser4 class/plugin interfaces that correspond very closely to the VFS class/plugin interfaces (the *_operations). In this case it might make good sense to lift those plugins from Reiser4 to VFS plugins.

(If the VFS class/plugin interfaces are subsets of some Reiser4 class/plugin interfaces, you could also probably do the lifting. The additional Reiser4 class/plugin interface parts could be added to the split off and added to the VFS objects via the generic pointer field [i.e., use it to attach the additional data and class/plugin pointers to the objects]. As this adds some pointer pain for the additional parts you may not really be gaining anything. Another possibly approach might be to just extend each of the VFS object structures by making them the first entry in more flushed out Reiser4 specific structures [the Reiser4 code better be doing the allocating in this case]. *grin*)

Later! -T

Signature · 4 lines
 Tyson Whitehead  ([email protected] -- WSC-)
 Computer Engineer                          Dept. of Applied Mathematics,
 Graduate Student- Applied Mathematics      University of Western Ontario,
 GnuPG Key ID# 0x8A2AB5D8                   London, Ontario, Canada
#127 lore raw

Re: reiser4 plugins

In reply to Lincoln Dale

--- Lincoln Dale <[email protected]> wrote:
snip

Quoted 19 lines

Nikita basically said as much in Message-ID:
<[email protected]> earlier in this thread:
"But it is not so. There _are_
plugins-in-the-VFS. VFS operates on
opaque
objects (inodes, dentries, file system types)
through interfaces:

{inode,address_space,dentry,sb,etc.}_operations. Every file system
back-end if free to implement whatever number
of these interfaces. And
the do this already: check the sources; even
ext2 does this: e.g.,
ext2_fast_symlink_inode_operations and
ext2_symlink_inode_operations.

Quoted 8 lines

This is exactly what upper level reiser4
plugins are for.

I guess that one of Christoph Hellwig's
complaints is that reiser4
introduces another layer of abstraction to
implement something that
already exists."

What reiserfs4 brings is file based plugins. Which is extremely useful and powerful. I don't want to see this go away.

I think it is the task of the linux community to generalize the vfs layer and not lock out reiserfs4 until that is done. reiserfs4 wants to keep a plugin id for each and every file. An additional filesystem layer is the traditional solution to achieve advanced features, but not an optimal solution in my opinion. Yes gnome, kde and perhaps cifs do it. But if instead they used file plugins a lot more could be shared.

Blush, I am not a file system expert

-Tim

__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com

#128 lore raw

Re: reiser4 plugins

Timothy Webster <[email protected]> wrote:

[...]

Quoted 3 lines

I think it is the task of the linux community to generalize the vfs layer and not lock out reiserfs4 until that is done.

No... the task of the Linux community is to make a better kernel.

You will have to do the work to convince it that the changes give a better kernel if included. That means the work to find out what they object to, solve the problem (if it can be solved), and try again.

Quoted 2 lines

reiserfs4 wants to keep a plugin
id for each and every file.

Good for you. Nobody else has shown any interest in that.

Quoted 3 lines

An additional filesystem
layer is the traditional solution to achieve advanced features, but not an optimal solution in my opinion.

Right. And what is said here is to /use/ said layer (VFS) or see how it can be changed to cater for your needs.

Quoted 1 line

Yes gnome, kde and perhaps cifs do it.

Gnome and KDE are userspace utilities, designed to run on several operating systems that do not have such filesystem plugins at all, plus for the foreseable future the mayority of Linux (file)systems won't have them either; so they probably won't ever use it for portability's sake. CIFS is a second class citizen AFAIU, as it exists for compatibility with legacy systems only. So none of the above is in any way compelling.

Quoted 4 lines

But if instead
they used file plugins a lot more could be shared.

Blush, I am not a file system expert

/me neither ;-)

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#129 lore raw

Re: reiser4 plugins

Quoted 9 lines · Horst von Brand

Timothy Webster <[email protected]> wrote:

[...]

Quoted 3 lines

I think it is the task of the linux community to generalize the vfs layer and not lock out reiserfs4 until that is done.

No... the task of the Linux community is to make a better kernel.

In general, isn't it better to first include modules providing
divergent but possibly interesting functionality (such as Reiser4) as an "optional" or "experimental" component, and then slowly re-factor desirable functionality into higher level facilities like the VFS?

One of the strengths of Linux (so far), is that it hasn't tried
too hard to "guess" what functionality might be the next "killer" kernel feature; if something is truly interesting, it will eventually find its way more deeply into the kernel. If not, it stays in an "optional" driver.

Quoted 9 lines

You will have to do the work to convince it that the changes give a better kernel if included. That means the work to find out what they object to, solve the problem (if it can be solved), and try again.

Quoted 2 lines

reiserfs4 wants to keep a plugin
id for each and every file.

Good for you. Nobody else has shown any interest in that.

The jffs2 filesystem may perhaps vary its functionality and
"on-flash" format wildly, depending on whether you use it with NOR or NAND flash. Not many people have shown interest in that; however, it is still an important feature to some (ie. ME).

This is an implicit analog of a plugin; Why force the VFS to take
note of how jffs2 happens to handle its formatting for NAND and NOR flash? Just let it use the correct "plugin"!

Virtually every non-trivial filesystem has the concept of varying
its behaviour, some implicitly (jffs2 NAND or NOR?), some explicitly (NFS mount options). Should we force them ALL to select ALL format variability via the VFS?

So what if reiser4 wants to have a plugin that converts all
english files to swahili or pig latin as an on-disk format, and it wants to call it a "plugin". So long as it doesn't interfere with the correctness of other filesystems, why prevent it from merging?

I believe that it would be a GREATER error to force VFS changes
supporting the generic idea of a "plugin", before proving that the concept has more general applicability. See if reiser4 lives or dies first, as an optional experimental filesystem. No one is forced to use it. In a couple of years, if it becomes an ignored "backwater" filesystem (like, say, coda?), then remove it from the mainline kernel. No one will notice...

Quoted 7 lines
Quoted 4 lines

An additional filesystem
layer is the traditional solution to achieve advanced features, but not an optimal solution in my opinion.

Right. And what is said here is to /use/ said layer (VFS) or see how it can be changed to cater for your needs.

If reiser4 wants to provide a usable interface (internally), to
select behaviour that is traditionally selected by choosing a completely separate filesystem, then why not let it? It harms *no one* -- forcing changes to the VFS before it is proven that this kind of functionality is even generally desirable would be a HUGE error!

Quoted 12 lines
Quoted 2 lines

Yes gnome, kde and perhaps cifs do it.

Gnome and KDE are userspace utilities, designed to run on several operating systems that do not have such filesystem plugins at all, plus for the foreseable future the mayority of Linux (file)systems won't have them either; so they probably won't ever use it for portability's sake. CIFS is a second class citizen AFAIU, as it exists for compatibility with legacy systems only. So none of the above is in any way compelling.

Quoted 2 lines

But if instead
they used file plugins a lot more could be shared.

The ext2 and ext3 filesystems support the same basic on-disk
format, as everyone knows. However, the selection of ext3 provides an on-disk journal, which (if the filesystem is then mounted as ext2), is ignored. You select the "journalling plugin" by mounting the fs as ext3. You turn it off by mounting it as ext2.

Reiser4 provides a way to do precisely this kind of selection of
variable on-disk formats by specifying a plugin.

I ask you -- if everyone in kernel-land is so convinced that you
should always select varying on-disk formats via the VFS, then *why* hasn't ext2/ext3 been merged into a single filesystem? Surely the "journalling" plugin of this filesystem is a prime candidate for selection via the VFS?

Here's the deal.

So Hans can be a butt-head. Get over it. Big projects attract
big egos. This is a good thing!

Reiser4 provides potentially very interesting functionality, some
of which may or may not (over time) be interesting to other FS implementers. At that time, it will be "abstracted" into the VFS. *AFTER* it has been proven safe and useful to do so!

Until then, including Reiser4 in the main-line kernel is
*completely* consistent with the "do no harm" philosophy that has let other filesystems and drivers have be included. No one running a "stable" server is going to let the "experimental" Reiser4 filesystem code anywhere near their kernel -- just like all other "experimental" code!

Others (like me), will use the 'md' drivers to provide the stable,
error-free media that ALL filesystems, including reiser4, assume, 'rsync' to provide for backups since it is clearly marked as "experimental", and continue to benefit from the improved atomicity and speed that reiser4 provides -- just as I have been doing for the last 18 months!

Signature · 8 lines
    -pjk

"If you will not fight when your victory will be sure and not too
costly, you may come to the moment when you will have to fight with
all the odds against you and only a precarious chance for survival.
There may even be a worse case. You may have to fight when there is no
hope of victory, because it is better to perish than to live as
slaves." -- Winston Churchill
#130 lore raw

Fwd: reiser4 plugins

Quoted 9 lines · Horst von Brand

Timothy Webster <[email protected]> wrote:

[...]

Quoted 3 lines

I think it is the task of the linux community to generalize the vfs layer and not lock out reiserfs4 until that is done.

No... the task of the Linux community is to make a better kernel.

In general, isn't it better to first include modules providing
divergent but possibly interesting functionality (such as Reiser4) as an "optional" or "experimental" component, and then slowly re-factor desirable functionality into higher level facilities like the VFS?

One of the strengths of Linux (so far), is that it hasn't tried
too hard to "guess" what functionality might be the next "killer" kernel feature; if something is truly interesting, it will eventually find its way more deeply into the kernel. If not, it stays in an "optional" driver.

Quoted 9 lines

You will have to do the work to convince it that the changes give a better kernel if included. That means the work to find out what they object to, solve the problem (if it can be solved), and try again.

Quoted 2 lines

reiserfs4 wants to keep a plugin
id for each and every file.

Good for you. Nobody else has shown any interest in that.

The jffs2 filesystem may perhaps vary its functionality and
"on-flash" format wildly, depending on whether you use it with NOR or NAND flash. Not many people have shown interest in that; however, it is still an important feature to some (ie. ME).

This is an implicit analog of a plugin; Why force the VFS to take
note of how jffs2 happens to handle its formatting for NAND and NOR flash? Just let it use the correct "plugin"!

Virtually every non-trivial filesystem has the concept of varying
its behaviour, some implicitly (jffs2 NAND or NOR?), some explicitly (NFS mount options). Should we force them ALL to select ALL format variability via the VFS?

So what if reiser4 wants to have a plugin that converts all
english files to swahili or pig latin as an on-disk format, and it wants to call it a "plugin". So long as it doesn't interfere with the correctness of other filesystems, why prevent it from merging?

I believe that it would be a GREATER error to force VFS changes
supporting the generic idea of a "plugin", before proving that the concept has more general applicability. See if reiser4 lives or dies first, as an optional experimental filesystem. No one is forced to use it. In a couple of years, if it becomes an ignored "backwater" filesystem (like, say, coda?), then remove it from the mainline kernel. No one will notice...

Quoted 7 lines
Quoted 4 lines

An additional filesystem
layer is the traditional solution to achieve advanced features, but not an optimal solution in my opinion.

Right. And what is said here is to /use/ said layer (VFS) or see how it can be changed to cater for your needs.

If reiser4 wants to provide a usable interface (internally), to
select behaviour that is traditionally selected by choosing a completely separate filesystem, then why not let it? It harms *no one* -- forcing changes to the VFS before it is proven that this kind of functionality is even generally desirable would be a HUGE error!

Quoted 12 lines
Quoted 2 lines

Yes gnome, kde and perhaps cifs do it.

Gnome and KDE are userspace utilities, designed to run on several operating systems that do not have such filesystem plugins at all, plus for the foreseable future the mayority of Linux (file)systems won't have them either; so they probably won't ever use it for portability's sake. CIFS is a second class citizen AFAIU, as it exists for compatibility with legacy systems only. So none of the above is in any way compelling.

Quoted 2 lines

But if instead
they used file plugins a lot more could be shared.

The ext2 and ext3 filesystems support the same basic on-disk
format, as everyone knows. However, the selection of ext3 provides an on-disk journal, which (if the filesystem is then mounted as ext2), is ignored. You select the "journalling plugin" by mounting the fs as ext3. You turn it off by mounting it as ext2.

Reiser4 provides a way to do precisely this kind of selection of
variable on-disk formats by specifying a plugin.

I ask you -- if everyone in kernel-land is so convinced that you
should always select varying on-disk formats via the VFS, then *why* hasn't ext2/ext3 been merged into a single filesystem? Surely the "journalling" plugin of this filesystem is a prime candidate for selection via the VFS?

Here's the deal.

So Hans can be a butt-head. Get over it. Big projects attract
big egos. This is a good thing!

Reiser4 provides potentially very interesting functionality, some
of which may or may not (over time) be interesting to other FS implementers. At that time, it will be "abstracted" into the VFS. *AFTER* it has been proven safe and useful to do so!

Until then, including Reiser4 in the main-line kernel is
*completely* consistent with the "do no harm" philosophy that has let other filesystems and drivers have be included. No one running a "stable" server is going to let the "experimental" Reiser4 filesystem code anywhere near their kernel -- just like all other "experimental" code!

Others (like me), will use the 'md' drivers to provide the stable,
error-free media that ALL filesystems, including reiser4, assume, 'rsync' to provide for backups since it is clearly marked as "experimental", and continue to benefit from the improved atomicity and speed that reiser4 provides -- just as I have been doing for the last 18 months!

--
-pjk

"If you will not fight when your victory will be sure and not too costly, you may come to the moment when you will have to fight with all the odds against you and only a precarious chance for survival. There may even be a worse case. You may have to fight when there is no hope of victory, because it is better to perish than to live as slaves." -- Winston Churchill

#131 lore raw

Re: reiser4 plugins

In reply to Perry Kundert
Quoted 4 lines · Perry Kundert

In general, isn't it better to first include modules providing
divergent but possibly interesting functionality (such as Reiser4) as an "optional" or "experimental" component, and then slowly re-factor desirable functionality into higher level facilities like the VFS?

The problem arises when the facility is something that is demonstrably borked when done in an optional way in one filesystem, and really needs to be done at the VFS level if it is to be done at all.

Quoted 3 lines

I ask you -- if everyone in kernel-land is so convinced that you
should always select varying on-disk formats via the VFS, then *why* hasn't ext2/ext3 been merged into a single filesystem?

Because the formats, although similar enough to be mostly compatible, are still different enough that merging them is difficult. There's some very subtle second-order effects, where the ext3 driver can do things in different orders or with different algorithms because it has a journal, when the ext2 code has to do things in a specific way because it has to *always* have things in a consistent enough state that fsck.ext2 can clean things up. So you end up with code that looks like:

	if (fs->journalled) {
		/* 500 lines of code for the ext3 case */
	} else {
		/* 300 lines of different code for ext2 */
	}

If you don't like that, then you can do this instead:

1) put ext2_do_whatever in ext2_whatever.c
2) put ext3_do_whatever in ext3_whatever.c

extern ext2_do_whatever();
extern ext3_do_whatever();

	if (fs-> journalled) {
		ext3_do_whatever();
	} else {
		ext2_do_whatever();
	}

In fact, I seem to remember Alan Cox answering this with "only about 10% of the code *wouldn't* end up like this" or similar...

Quoted 3 lines

Surely the
"journalling" plugin of this filesystem is a prime candidate for selection via the VFS?

To be doing "journalling" at the VFS level implies that a journal is something that makes sense at the VFS level - that it's basically filesystem independent, which is most certainly *not* true - the notations an XFS journal needs to make to indicate which blocks were just removed from the free-block structure are quite different from what ext3 needs to record.

Note that journalling is neither an attribute of the actual data, or of
the user-visible metadata (inode contents, etc). The only things that
care about the journalling format/etc are the filesystem driver, the mount
command, and the mkfs/fsck commands. As such, it's a file system issue,
not a VFS issue.

For a good example of why this is so, go back and read the recent discussion of what happens to flash memory filesystems mounted with 'sync' - this was a case of the VFS doing "journalling by flushing" without consulting the low-level drivers....

#132 lore raw

Re: reiser4 plugins

Quoted 10 lines · On 6/24/05, [email protected] <[email protected]> wrote:
Quoted 4 lines · Perry Kundert

In general, isn't it better to first include modules providing
divergent but possibly interesting functionality (such as Reiser4) as an "optional" or "experimental" component, and then slowly re-factor desirable functionality into higher level facilities like the VFS?

The problem arises when the facility is something that is demonstrably borked when done in an optional way in one filesystem, and really needs to be done at the VFS level if it is to be done at all.

OK, fair enough. The file-as-directory stuff, which introduces
VFS-incompatible issues, was turned off. It requires VFS changes.

The remaining plugin architecture, as far as I understand, deals
in the on-disk structure of the FS -- just like journals. Encryption, Compression, and the like.

So, what you are saying, is -- so long as the "plugins" do stuff
that deals in how reiser4 slings bits back and forth to the disk, you're OK, right?

Quoted 7 lines
Quoted 4 lines

I ask you -- if everyone in kernel-land is so convinced that you
should always select varying on-disk formats via the VFS, then *why* hasn't ext2/ext3 been merged into a single filesystem?

Because the formats, although similar enough to be mostly compatible, are still different enough that merging them is difficult.

So, what you are saying is: if reiser4 wants to provide
variability in on-disk format, so long as it implements it using *multiple different filesystems* (eg reiser4-cryptcompress, reiser4-whatever) -- just like ext2 vs. ext3 -- then you are OK with it? But, if they are implemented as "plugins", so that the ONE reiser4 filesystem can modulate its behaviour based on what the on-disk format says, that you are NOT OK with it?

And this makes sense, why?

Quoted 7 lines

There's some very
subtle second-order effects, where the ext3 driver can do things in different orders or with different algorithms because it has a journal, when the ext2 code has to do things in a specific way because it has to *always* have things in a consistent enough state that fsck.ext2 can clean things up. So you end up with code that looks like:
...

Don't get me wrong -- I'm not saying that ext2 and ext3 shouldn't
be separate file systems. However, if they were designed from the start so that the ONE (say) ext23 filesystem could look at its on-disk format, notice that the data specified the "journal" plugin, and implement the correct behaviour -- that this would be "bad"?

Because I can envision an ext23 filesystem that is just like
reiser4, that does exactly that -- implements its variable behaviour via a "journal" plugin.

So, if it did so, would you be OK with it? As long as it wasn't
called reiser4?

I really don't mean to sound sarcastic -- it just sounds like
there are "other" issues at work here -- like "Hans is a Butt-Head, so I want to reject reiser4's plugin design for modulating its behviour, no matter what".

Quoted 9 lines
Quoted 3 lines

Surely the
"journalling" plugin of this filesystem is a prime candidate for selection via the VFS?

To be doing "journalling" at the VFS level implies that a journal is something that makes sense at the VFS level - that it's basically filesystem independent, which is most certainly *not* true - the notations an XFS journal needs to make to indicate which blocks were just removed from the free-block structure are quite different from what ext3 needs to record.

Sort of like crypt/compress, etc. -- each filesystem may or may
not do this, but it just doesn't matter to the VFS -- by the time data hits the VFS it isn't encrypted or compressed. So, why not let the FS say "hey, I happen to see that the on-disk format says it uses cryptcompress -- I'll dispatch the right methods from this aptly-named "plugin"...

Quoted 7 lines

Note that journalling is neither an attribute of the actual data, or of
the user-visible metadata (inode contents, etc). The only things that
care about the journalling format/etc are the filesystem driver, the mount
command, and the mkfs/fsck commands. As such, it's a file system issue,
not a VFS issue.

Quoted 7 lines

For a good example of why this is so, go back and read the recent discussion of what happens to flash memory filesystems mounted with 'sync' - this was a case of the VFS doing "journalling by flushing" without consulting the low-level drivers....

OK, so far it seems like we are actually agreeing -- the stuff
that gets done via reiser4 "plugins" actually doesn't have anything to do with the VFS, and it shouldn't be there. So long as reiser4 presents a VFS-sensible, VFS-consistent heirarchy of stuff that "looks like" files and directories to the VFS, then we're OK with it?

Whether reiser4 uses "plugins", or ESP, or whatever to decide what
behaviour it implements in order to produce this VFS-consistent interface, then that's OK, right?

Signature · 1 line
    -pjk
#133 lore raw

Re: reiser4 plugins

Quoted 2 lines · Perry Kundert

OK, fair enough. The file-as-directory stuff, which introduces
VFS-incompatible issues, was turned off. It requires VFS changes.

Mind you, I still think that sounds *interesting*, but it *has* to happen at the VFS level. (And if *that* doesn't force a 2.7 fork to happen, nothing will :)

Quoted 7 lines

The remaining plugin architecture, as far as I understand, deals
in the on-disk structure of the FS -- just like journals. Encryption, Compression, and the like.

So, what you are saying, is -- so long as the "plugins" do stuff
that deals in how reiser4 slings bits back and forth to the disk, you're OK, right?

Right - once the VFS hands the call off to reiser4, you're on your own as far as I'm concerned..

Quoted 9 lines

So, what you are saying is: if reiser4 wants to provide
variability in on-disk format, so long as it implements it using *multiple different filesystems* (eg reiser4-cryptcompress, reiser4-whatever) -- just like ext2 vs. ext3 -- then you are OK with it? But, if they are implemented as "plugins", so that the ONE reiser4 filesystem can modulate its behaviour based on what the on-disk format says, that you are NOT OK with it?

And this makes sense, why?

You misread that - my point was that ext2 and ext3 may look similar, but they're sufficiently divergent that trying to create one driver that handles both results in an ugly driver, thus the split...

Quoted 5 lines

Don't get me wrong -- I'm not saying that ext2 and ext3 shouldn't
be separate file systems. However, if they were designed from the start so that the ONE (say) ext23 filesystem could look at its on-disk format, notice that the data specified the "journal" plugin, and implement the correct behaviour -- that this would be "bad"?

Well, if they *had* been, it would be a different story. And there's stuff in ext3 (see the "-O feature" section of 'man tune2fs') that *does* do the sort of thing that you're proposing. It's just that the ext2 codebase doesn't fit in well for historical reasons.

Quoted 6 lines

Because I can envision an ext23 filesystem that is just like
reiser4, that does exactly that -- implements its variable behaviour via a "journal" plugin.

So, if it did so, would you be OK with it? As long as it wasn't
called reiser4?

No, I'd be perfectly happy with a reiser4 that had a 'tunereiser --enable-plugin=' that had the same sort of format-altering semantics that 'tune2fs -O' has.

For bonus points, design a system that stores the plugin *in the file system* (probably need to have a bytecode interpreter for this). Then you eliminate the "can't mount if the kernel can't insmod the plugin" issue ;)

Quoted 4 lines

I really don't mean to sound sarcastic -- it just sounds like
there are "other" issues at work here -- like "Hans is a Butt-Head, so I want to reject reiser4's plugin design for modulating its behviour, no matter what".

I've never actually met Hans, so I don't know if he really *is* a butt-head or not. And I usually at least *try* to phrase it more like "this proposal is a non-starter that only a butt-head could continue to support, because.." :)

Of course, I myself have been called a butt-head on numerous occasions, because I'm convinced there's a right and wrong way to design something. ;)

Quoted 9 lines

OK, so far it seems like we are actually agreeing -- the stuff
that gets done via reiser4 "plugins" actually doesn't have anything to do with the VFS, and it shouldn't be there. So long as reiser4 presents a VFS-sensible, VFS-consistent heirarchy of stuff that "looks like" files and directories to the VFS, then we're OK with it?

Whether reiser4 uses "plugins", or ESP, or whatever to decide what
behaviour it implements in order to produce this VFS-consistent interface, then that's OK, right?

Right - not that *my* opinion counts for tons on LKML, and any *other* stylistic/design faults are a separate issue. :)

#134 lore raw

Re: reiser4 plugins

Quoted 5 lines · [email protected] wrote:

Right - once the VFS hands the call off to reiser4, you're on your own

as far as I'm concerned..

Well, that is all I ask for, and Christophe and company disagree..... Happy to abstract it more into VFS after the merge if others want it though....

#135 lore raw

Re: reiser4 plugins

Quoted 10 lines · Hans Reiser
Quoted 5 lines · [email protected] wrote:

Right - once the VFS hands the call off to reiser4, you're on your own

as far as I'm concerned..

Well, that is all I ask for, and Christophe and company disagree..... Happy to abstract it more into VFS after the merge if others want it though....

Unfortunately, the only *realistic* path I can see at the moment is to strip it down to those reiser4 features that don't require any VFS help/changes to do, and pursue VFS changes as a separate issue entirely.

Also, just as a personal note - talking about a "reiser4 plugin that does ext3 backend storage" doesn't help the cause. What you *should* be trying to sell:

1) a reiser4 storage backend with backend plugins that follows the VFS conventions.
2) a vastly enhanced VFS that has VFS plugins and calls reiser4 and ext3 backends.

Approaching it that way will make people think that (a) you really *do* understand the VFS/FS layering and (b) are willing to compromise to get things done.. ;)

#136 lore raw

Re: reiser4 plugins

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

[email protected] wrote:
[...]

Quoted 2 lines

Also, just as a personal note - talking about a "reiser4 plugin that does ext3 backend storage" doesn't help the cause. What you *should* be trying to sell:

To be fair, I brought that up. I don't remember if Hans endorsed it, but I don't think so.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrzEB3gHNmZLgCUhAQKltBAAn4+NAPUwiIem30RtTuE+BskPY8rhPuCq ExA/lsbC87MffqEHne5xVMAXGiSNfgb23mab3oQiyvnYy2zOgs0rHLchyAinpw/u q/CoZdi8BcSUpR5jlTYf9BVExJ2IPQ3C/MDDTt/LNqOWosipuiauqlKZk9Mvr35+ nKaA0BRhNbDG9E21ukMGR6/NauJJQwaHR0HE3rxwEyJbtJmpQHX2/YuYv5RhAQFH qq0U/67SmIfGBlhxk2fq6OnMR39lvs6kMt2rt6wqPQi9hi8jeAyBh7OXxfjoqpFF S6oI81dL+HyR1aqwyIK3rR+QBIH8iCc0wU8IaRbWws1v7SD+PUJcE9TmgWkRpxep 5bzOYApFsqKTbcCAiTUbPzVeNW1iglNvg9+tCS6cMJ6pRuEAenmJJeRQNRvho1KQ 7PABQPCzV0DPmX11vsOMJNfxmECKPchpglWtxjq0P6xM9XUxhrAy/S7RxvWlYe/q ckt046jFYzZHq29AZxUNRcruqCBk/Px7ZhI6QVEIHOx0J5SM6hBfKSmkj6WYWLKk o1l/vgaI5nb24zGnSueSSLn2utrZdQ8hRNGaCEbogMek90ltO4Zv0BEamcBDoDof 2s+gk4sBQ/muUPJppJ/a317cyV8BCKLknfT1IPa8G9dePWziKbcUNwA2MzR/EzNl VywXtawu9uA=
=P9Qs
-----END PGP SIGNATURE-----

#137 lore raw

Re: reiser4 plugins

In reply to [email protected]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

[email protected] wrote:
[...]

Quoted 11 lines
Quoted 7 lines

Because I can envision an ext23 filesystem that is just like
reiser4, that does exactly that -- implements its variable behaviour via a "journal" plugin.

So, if it did so, would you be OK with it? As long as it wasn't
called reiser4?

No, I'd be perfectly happy with a reiser4 that had a 'tunereiser --enable-plugin=' that had the same sort of format-altering semantics that 'tune2fs -O' has.

The question is whether you'd be happy if the kernel did that also. mount -t reiser4 -o enable_plugin=

I mean, I doubt the kernel team cares at all how the userland utilities work. After all, some of them seem quite happy to delegate ALL of the cool Reiser4 "plugin" things to userspace...

And whether it's true or not, implying an anti-reiser conspiracy is not very diplomatic...

Quoted 3 lines

For bonus points, design a system that stores the plugin *in the file system* (probably need to have a bytecode interpreter for this). Then you eliminate the "can't mount if the kernel can't insmod the plugin" issue ;)

Right now, "plugin" isn't quite accurate, as plugins can't even be separate modules, and to install a new plugin, you have to recompile the kernel, or at least the reiser4 module.

But, before anyone gets the wrong idea, they are still somewhat "plugins" because you don't have to _use_ any plugins you don't like. If you don't use them, they will just eat kernel RAM -- and a fixed amount of kernel RAM, not at all relative to how big your FS is.

I actually like the bytecode idea, except that then you need a bytecode interpreter (or JIT compiler) in the kernel. I like the idea of that, but I'm sure other people would be screaming bloody kernel bloat.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrzGKXgHNmZLgCUhAQKJag/9GxPnsNpaw3Z7xOoiY9kZVoa6vLWALH7T Xher++7289Ra1mYznInBTBjaBdEkk/AWRLzM6Q5Hl4F3d96HzweHsBTgOg/dtscH R3vTpaOQUxbOJWXBrsX/Jw/Q4NJHLDfbr3fg4vsYEgtzzKWkeIOCJgliW5XAB8+W 2Ry2gMj7QxZoQWeWrOPqAEtDudIHZDFFtg0FzWaZw9d0G/jIrAAi3gcGxqIZ4iK+ vNGnzn9K1Sk01TEclvRwh/+5dRMs/liEAlYxa03FB8Lx/2jj9v0WJBnXiPuxztsv hUtIcd9iA4S3AdYXefxHYbSuV2HO7DKyR+WJriyNqk5cpTFecaQ3BqaBCsiPaKlT IrPsm8R3xmsjZ6yB3qIELpe2wg0knhnbwEWMmFnmWUKrMrzpsrI2fcvxlOdcTZKZ 2kRumVNpZ6E0ZhIs5PxhkfBGoVPYMga4hrWNFCW6wuaFwskEPW+hx8su6amqAr1a AJwndr/QLVOo5f20vOk0eOvhjhLW7iZPLPkLJBw+1BnyS4cZuEg7N3pkZsDvKPCe +qqBPv2wGCzYQWlQDTsYriZA7y6UQLmcED/Y2GBlLYQzm8pq20w715/N3LD0G+GK 4lFwSV+hDFoVn5kSJNemPYhgfDT4hAOQyMHTKKY02XnNynAfT4TsWZgzjjSxWjSg QrueuVgBcRs=
=8xv0
-----END PGP SIGNATURE-----

#138 lore raw

Re: reiser4 plugins

In reply to Hans Reiser
Quoted 6 lines · Hans Reiser

Regarding trust, Christophe comes out the gate using the words "useless abstraction layer" that happens to be a core feature of our design, demanding we cut it out, and not really understanding it or recognizing any functionality it provides, and you can't really expect me to respect this, can you?

Quoted 6 lines

Now, if his target is reduced to whether we can eliminate a function indirection, and whether we can review the code together and see if it is easy to extend plugins and pluginids to other filesystems by finding places to make it more generic and accepting of per filesystem plugins, especially if it is not tied to going into 2.6.13, well, that is the conversation I would have liked to have had.

Have I missed the posting with analysis of changes in locking scheme and update of proof of correctness? If so, please give the message ID.

_That_ had been the major showstopper for any merges, IIRC.

#139 lore raw

Re: reiser4 plugins

Quoted 6 lines · Al Viro

Have I missed the posting with analysis of changes in locking scheme and update of proof of correctness? If so, please give the message ID.

_That_ had been the major showstopper for any merges, IIRC.

Ah, the prince of helpfulness has arrived.

Yes, as I remember, last time with V3 you announced that there were race conditions that we needed to fix if V3 was to be merged, you would not tell us what they were when asked, Linus merged us anyway, you never did tell us what they were, later vs fixed some race conditions but I have no idea if they were the same ones you found, oh well, getting rid of bugs never was your objective was it? Does V3 still have those race conditions you spoke of?

Proof of correctness, is that where we check and see if the filesystems all mount/unmount before checking in code changes to the stable release branch? Oh dear, that was unkind of me.

Ok, sure, define what you want in the way of a proof of correctness and an analysis. Is this a new VFS tradition? Is it documented anywhere? Are there tools for it? Probably I should ask Zam if we already did it too..... Zam?

Hans

#140 lore raw

Re: reiser4 plugins

Quoted 11 lines · Hans Reiser
Quoted 6 lines · Al Viro

Have I missed the posting with analysis of changes in locking scheme and update of proof of correctness? If so, please give the message ID.

_That_ had been the major showstopper for any merges, IIRC.

Ah, the prince of helpfulness has arrived.

Yes, as I remember,

Kindly put some efforts into remembering the thread that contains e.g.
http://marc.theaimsgroup.com/?l=linux-kernel&m=109347094309283&w=2

If that work (summary: introduction of hybrid objects invalidates the existing locking scheme for directories and that had lead at least to several user-exploitable deadlocks described in details in the same thread; current proof of correctness is in the tree, see Documentation/filesystems/Directory-Locking.txt and at the very least it needs to be updated) had been done - please, give the message ID of posting with such update. If not - please, arrange getting it done.

#141 lore raw

Re: reiser4 plugins

Quoted 13 lines · Al Viro
Quoted 11 lines · Hans Reiser
Quoted 6 lines · Al Viro

Have I missed the posting with analysis of changes in locking scheme and update of proof of correctness? If so, please give the message ID.

_That_ had been the major showstopper for any merges, IIRC.

Ah, the prince of helpfulness has arrived.

Yes, as I remember,

Quoted 8 lines

Kindly put some efforts into remembering the thread that contains e.g.
http://marc.theaimsgroup.com/?l=linux-kernel&m=109347094309283&w=2

If that work (summary: introduction of hybrid objects invalidates the existing locking scheme for directories and that had lead at least to several user-exploitable deadlocks described in details in the same thread; current proof of correctness is in the tree, see Documentation/filesystems/Directory-Locking.txt and at the very least

gaaack... Sorry - should've proofread better -
s/Directory-Locking.txt/directory-locking/ in the above.

Quoted 7 lines

it needs to be updated) had been done - please, give the message ID of posting with such update. If not - please, arrange getting it done. -
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#142 lore raw

Re: reiser4 plugins

In reply to Al Viro
Quoted 24 lines · Al Viro
Quoted 7 lines · Hans Reiser
Quoted 3 lines · Al Viro

Have I missed the posting with analysis of changes in locking scheme and update of proof of correctness? If so, please give the message ID.

Quoted 10 lines
Quoted 5 lines

_That_ had been the major showstopper for any merges, IIRC.

Ah, the prince of helpfulness has arrived.

Yes, as I remember,

Kindly put some efforts into remembering the thread that contains e.g.
http://marc.theaimsgroup.com/?l=linux-kernel&m=109347094309283&w=2

Quoted 11 lines

If that work (summary: introduction of hybrid objects invalidates the existing locking scheme for directories and that had lead at least to several user-exploitable deadlocks described in details in the same thread; current proof of correctness is in the tree, see Documentation/filesystems/Directory-Locking.txt and at the very least it needs to be updated) had been done - please, give the message ID of posting with such update. If not - please, arrange getting it done.

We disabled metafiles until we can fix this. With no metafiles, the issue does not exist.

Thanks for your assistance with finding that problem.

Hans

#143 lore raw

Re: reiser4 plugins

In reply to Hans Reiser
Quoted 4 lines · Hans Reiser
Quoted 2 lines

In which case the features belong in the VFS as all those with experience and kernel contributions have been arguing.

So you fundamentally reject the prototype it in one fs and then abstract it to others development model?

More fundamentally - prototype things *out* of the main kernel. If everyone was doing their prototyping in kernel Andrew Morton would by now be a team of about 25

Alan

#144 lore raw

Re: reiser4 plugins

Quoted 4 lines · Alan Cox

More fundamentally - prototype things *out* of the main kernel. If everyone was doing their prototyping in kernel Andrew Morton would by now be a team of about 25

This is going semi off-topic but how then do you justify regression that's apparently confessed? :)

It appears -mm is more of a prototype tree than a development tree; I remember Con Kolivas cursing (with SMP nice support) that the interface is too lively to keep supporting one patch. Unfortunately I can't find the original post I'm thinking about but
http://lkml.org/lkml/2005/5/16/68 says essentially the same thing.

There's also my all-time favorite, http://lkml.org/lkml/2005/3/14/4

The lack of QA seems appalling here, and I'm sure Reiser has had to do more of that for DARPA than most linux kernel hackers around.

What I'm saying here is isn't it a bit hypocritic to ramble on that we need real assurances for this to work, community proof and bugfixes on the list mean nothing, when other core systems seem to be much looser in this respect?

Sure, "other people merge design-breakers and bugs" is NOT a justification for merging Reiser4, of course, but Reiser4's track record has vastly cleaned itself up.

Most bug reports come from broken hardware or unsupported patches or old code. Just have Namesys swear they won't introduce design changes until the userland utils are available, and won't do it at all after EXPERIMENTAL has been removed.
They already did this with changes that require mkfs :>

Here's a real suggestion, for LKML et al, dropping the sarcastic mode.

0. Namesys addresses the code beautification reqs mentioned here earlier.

1. Merge Reiser4 sans metas into 2.6.13.
2. Namesys can have a separate metas patch for testing.
3. Gradually merge Reiser4 architecture into VFS and if this really requires a 2.7, as (iirc) Valdis Kletnieks suggested, make it so. Might do the rest of the kernel some good too.

The above is a lame compromise; I'd much rather have the meta system in, as is dropping the stupid name, ..metas/ is better, and dropping "meta files can have meta data, ie. endless recursion in ..metas/

Then it can be merged into the VFS as needed, but if there truly are issues left, I realize it can't probably be fixed fast enough.

Having metas on by default probably leads to a lot of bug reports, which get fixed, and at the same time makes VFS-merging easier, when metas-related bug issues are fixed in VFS.

My two grumpy cents.

Signature · 1 line
mjt
#145 lore raw

Re: reiser4 plugins

Quoted 2 lines · Markus Törnqvist

I can't find the original post I'm thinking about but
http://lkml.org/lkml/2005/5/16/68 says essentially the same thing.

The scheduler is being improved for better behaviour on complex topologies like multi core + NUMA and multi level NUMA systems. If Con's work had gone in first, then conversely these improvements would have had to wait.

Quoted 2 lines

There's also my all-time favorite, http://lkml.org/lkml/2005/3/14/4

What's wrong with that? The slowdown is due to the workload becoming disk bound. The reasons are still not entirely clear, but I don't think it is a recent (ie. 2.6) regression (or even a regression at all IIRC).

Quoted 3 lines

The lack of QA seems appalling here, and I'm sure Reiser has had to do more of that for DARPA than most linux kernel hackers around.

And what QA would you have preferred?

I think if you are resorting to bringing up all time favourite blunders when trying to justify Reiser4 being included, then that is a sign right there that something is fundamentally wrong (if not with the code, then with your line of thought0

And note my email has nothing to do with any *real* argument for or against R4.

Thanks,
Nick

Send instant messages to your online friends http://au.messenger.yahoo.com

#146 lore raw

Re: reiser4 plugins

Quoted 5 lines · Nick Piggin

The scheduler is being improved for better behaviour on complex topologies like multi core + NUMA and multi level NUMA systems. If Con's work had gone in first, then conversely these improvements would have had to wait.

Or be merged in later.

The problem is, why do the interfaces have to live so much that examples like this come to my ears (or eyes ;) all the time?

I hate to say this without digging out any URLs, but one friend of mine says he has a very hard time doing any networking code because it's too labile. Maybe that's being embettered for something else too?

Or the other friend who curses that the networking code is just crap and basically has to rewrite the code to get it working. Yes, I've tried to get these guys to submit their code, but they argue back that no one wants to see it.

Quoted 6 lines
Quoted 1 line

There's also my all-time favorite, http://lkml.org/lkml/2005/3/14/4

What's wrong with that? The slowdown is due to the workload becoming disk bound. The reasons are still not entirely clear, but I don't think it is a recent (ie. 2.6) regression (or even a regression at all IIRC).

It's not that easy.

So let's say the scheduler slowdown is a temporary situation to adapt it to multicore+NUMA.
I assume that's what you mean, by having the kernels on par with 2.6.10 and the above paragraph.

Why on earth did the slowdown ever get merged and we have to wait for it to be on par with some older version?

Maybe the multicore+NUMA guys don't think it's a regression, hell, it may be better for them at the cost of the embedded (or desktop or whoever) guys.

Still my initial reaction is "if a patch slows things down, revert it. If it didn't do anything, keep reverting until you have the original situation."
It's also my reaction after the initial one :)

Sure, 2.4 and others did also have issues, so you couldn't automatically assume each and every new kernel would be better than the last, but it seems nowadays it's also hard to predict even the trend.

But to be fair, Linux is not a disaster yet and you guys are doing good work, despite things looking very shaky at times and my temper being more and more easily flared nowadays at signs of trouble :)

Quoted 4 lines

I think if you are resorting to bringing up all time favourite blunders when trying to justify Reiser4 being included, then that is a sign right there that something is fundamentally wrong (if not with the code, then with your line of thought0

Myeah, I know, this is not helping the Reiser4 cause _as_such_.

What I'm saying is, Reiser4 could be merged as it looks like a lot of other stuff is merged too, which may not be very tested.

Reiser4 at least is tested.

We can skip the parts that appear to suck.

Quoted 2 lines

And note my email has nothing to do with any *real* argument for or against R4.

I think this could be further split into an argument about the development model :P

Having a separate dev tree would keep the majority happy, as things are "good enough for most people" and after all has been evened out in the dev tree, a new stable tree can be launched.

This particular new file system should not be too drastic a thing to merge into a stable kernel, without the instability- inducing factors, and I think in the old model, all this would have been fleshed out in the dev tree automagically.

Please do correct me if I'm wrong :)

Thanks!

Signature · 1 line
mjt
#147 lore raw

Re: reiser4 plugins

Quoted 10 lines · Markus Törnqvist
Quoted 4 lines · Nick Piggin

The scheduler is being improved for better behaviour on complex topologies like multi core + NUMA and multi level NUMA systems. If Con's work had gone in first, then conversely these improvements would have had to wait.

Or be merged in later.

It _was_ merged later. Con was talking about Andrew's -mm tree.

Quoted 3 lines

The problem is, why do the interfaces have to live so much that examples like this come to my ears (or eyes ;) all the time?

The scheduler interfaces are probably among the most stable in the kernel. Con was talking about internal implementation.

Quoted 5 lines

I hate to say this without digging out any URLs, but one friend of mine says he has a very hard time doing any networking code because it's too labile. Maybe that's being embettered for something else too?

It seems there are plenty of people who can do networking code just fine.

Quoted 5 lines

Or the other friend who curses that the networking code is just crap and basically has to rewrite the code to get it working. Yes, I've tried to get these guys to submit their code, but they argue back that no one wants to see it.

I get the feeling your friend is making up some tall tales if he says he rewrote the networking code to be much better than it is now. But I can guarantee him that if he has it will be of great interest to the network developers.

Quoted 9 lines
Quoted 7 lines
Quoted 2 lines

There's also my all-time favorite, http://lkml.org/lkml/2005/3/14/4

What's wrong with that? The slowdown is due to the workload becoming disk bound. The reasons are still not entirely clear, but I don't think it is a recent (ie. 2.6) regression (or even a regression at all IIRC).

Quoted 6 lines

It's not that easy.

So let's say the scheduler slowdown is a temporary situation to adapt it to multicore+NUMA.
I assume that's what you mean, by having the kernels on par with 2.6.10 and the above paragraph.

I'm pretty sure it isn't a scheduler slowdown, but rather the system is getting disk bound. I think the reporter has performance back to 2.6.10 levels - it is actually something I still have to follow up on, but it is difficult because the reporter isn't able to share his code, although he's otherwise very helpful.

Quoted 4 lines

Why on earth did the slowdown ever get merged and we have to wait for it to be on par with some older version?

If slowdowns do get merged, it is generally because they either haven't been reported until being merged or they make an accepted tradeoff. Not that they haven't been tested.

Let's just get this straight, no amount of QA other than releasing a kernel and asking people to use it is going to find all the bugs that will be encountered.

Quoted 3 lines

Please do correct me if I'm wrong :)

I mostly agree with you apart from your specific examples I think we could do things better, and most of us have made some big blunders, However there is just no way that bringing any of that up will help Reiser4 being merged.

Send instant messages to your online friends http://au.messenger.yahoo.com

#148 lore raw

Re: reiser4 plugins

Quoted 2 lines · Nick Piggin

The scheduler interfaces are probably among the most stable in the kernel. Con was talking about internal implementation.

Hmm, I'm by no means a kernel developer, but looking at the patches, it looked like the functions were just renamed and maybe something else was changed.

I may be totally wrong here, and I apologize in advance if I am, but it seems to me like there would be no real need in changing some syntax?

Quoted 4 lines

I get the feeling your friend is making up some tall tales if he says he rewrote the networking code to be much better than it is now. But I can guarantee him that if he has it will be of great interest to the network developers.

Yeah, I just talked this over with him.

[
IF THIS IS BULLSHIT, LET ME KNOW AND I'LL CORRECT HIM ;) POINTERS TO RELEVANT DOCUMENTATION ALSO WELCOME ;)
]

His grudge was that conntrack was used everywhere, not like in 2.4, with a slow lookup in proc, that made it unusable for large nat systems.

They wrote some reimplementation of the lookup part, which they didn't release.

Then they figured out how to capture the packets before reaching conntrack and doing some mojo on them and reinjecting them somewhere later or bypassing conntrack altogether.

However, his opinions were even less motivated than mine, it seems, as he didn't have anything except some ipfilter archives he read, where they cursed the Linux way of doing things.

I haven't looked all too deeply into the network part, though, as I'm starting to get lazy and like to have a separate, dedicated piece of hardware do the network stuff for me...

So maybe I should just retract my comments and look into every possible subsystem's development and see for myself, which I don't have time for :)

Quoted 6 lines

I'm pretty sure it isn't a scheduler slowdown, but rather the system is getting disk bound. I think the reporter has performance back to 2.6.10 levels - it is actually something I still have to follow up on, but it is difficult because the reporter isn't able to share his code, although he's otherwise very helpful.

It's just that for cases like these testing would be great.

I'm not really sure how to do it, though, as it would require a lot of hardware and time and someone who can write a decent test suite.

If anyone has links to good test suites, I'm all ears :)

So, any chances of something being messed up in the disk part of the code?

...Too bad I just heard a big slowdown-in-disk-i/o curse was caused by dma falling off, for whatever reason (bug?) so that can't be used as forensics either :P

Quoted 7 lines

If slowdowns do get merged, it is generally because they either haven't been reported until being merged or they make an accepted tradeoff. Not that they haven't been tested.

Let's just get this straight, no amount of QA other than releasing a kernel and asking people to use it is going to find all the bugs that will be encountered.

Would you think a separate dev tree would help here?

I've also talked long with a guy from one of our major customers who says he's been losing faith in Linux over the past year, as his boxes are going oops or panicking every once in a while.

The problem is that their system is too redundantly set up for a single oops or panic to matter much, so I haven't convinced them to report bugs. They'd rather boot the boxes and curse.

He just figured it odd that this started happening after the new dev model, but the timelines are too granular for a real correlation. Just a strong gut feeling, and that's the kind of FUD that makes people look to *BSD, and a gut feeling that may have some merit to it, although it could never be proven. So don't take that as a flame or anything, just an observation :P

Sure it must be difficult developing the kernel, if it were easier, I'd probably be at it too, but people don't like not being well informed of accepted tradeoffs, as they might not accept said tradeoff, and tracking what caused it may be too much.

A lot of people out there don't like vendor kernels either, with 65536 backported patches and the 65537th voids your warranty.

It's also probably very difficult to do analytical QA before submitting code, as bugs ALWAYS run through.

But my gut tells me this -mm thing may not be the optimal way, as people are more scared of -mm, where some things may or may not work, than a separate dev tree, of which kerneltrap or whoever can post "this is severely broken, test if you like" and "this seems to be going mostly strong now, test it on your everyday desktop"

A lot of complaints seem to come from people who use patchsets with half-assed backports and experimental code, having a separate tree may drive these people to use the experimental stuff knowing what it is, instead of believing their patch set is as good as vanilla.

Quoted 4 lines

I mostly agree with you apart from your specific examples I think we could do things better, and most of us have made some big blunders, However there is just no way that bringing any of that up will help Reiser4 being merged.

Myeah, if I try to keep it a bit more on topic, I'm just saying that Reiser4 (sans metas) has been heavily tested, and now that the Namesys guys are apparently beautifying the code, we should be good with everything.

Still, this is an instinct, but it seems like the bigger VFS overhauls would be easier to do in an official dev tree with a big note saying "THIS WILL BE BROKEN FOR A MONTH."

If this was the case, so bad that people couldn't develop anything with the dev tree, it would probably be easier to maintain a rollback-vfs-changes patch than constantly keeping in sync with -mm.

That method could be used for every system undergoing extensive work that shows up in other systems.

The difference may not be big, but it may be enough.

Thanks!

Signature · 1 line
mjt
#149 lore raw

Re: reiser4 plugins

In reply to Markus Törnqvist

On 6/27/05, Markus Törnqvist <[email protected]> wrote:
[...]

Quoted 11 lines

I hate to say this without digging out any URLs, but one friend of mine says he has a very hard time doing any networking code because it's too labile. Maybe that's being embettered for something else too?

Or the other friend who curses that the networking code is just crap and basically has to rewrite the code to get it working. Yes, I've tried to get these guys to submit their code, but they argue back that no one wants to see it.

[...]

I'm pretty damn sure the relevant maintainers (and a bunch of people on LKML and the netdev lists in general) would like to see patches that improves their code.
If these friends of yours are sitting on patches that make massive improvements to the code and they are not submitting patches then they are not exactely helping (and I'd suspect them to just be full of BS)
- they should get their code merged instead of having to maintain it themselves out-of-tree for ever - let the rest of us bennefit as well. It's my experience that if you can explain the problems your patch fix and explain well why the fix is sane, then getting your patches merged doesn't have to be hard.

Signature · 3 lines
Jesper Juhl <[email protected]>
Don't top-post  http://www.catb.org/~esr/jargon/html/T/top-post.html
Plain text mails only, please      http://www.expita.com/nomime.html
#150 lore raw

Re: reiser4 plugins

In reply to Markus Törnqvist
Quoted 3 lines · Markus Törnqvist

Sure, "other people merge design-breakers and bugs" is NOT a justification for merging Reiser4, of course, but Reiser4's track record has vastly cleaned itself up.

The discussion is about merging from -mm, not into -mm. The merge into -mm isn't at all under discussion so I don't the relevance of the comparison.

Quoted 7 lines

0. Namesys addresses the code beautification reqs mentioned here earlier.

1. Merge Reiser4 sans metas into 2.6.13.
2. Namesys can have a separate metas patch for testing.
3. Gradually merge Reiser4 architecture into VFS and if this really requires a 2.7, as (iirc) Valdis Kletnieks suggested, make it so. Might do the rest of the kernel some good too.

0 and 1 look like the first right steps to take to me as well. That will allow people to use reiser4 and give it a good hammering see what comes out and how it benches in real life.

Alan

#151 lore raw

Re: reiser4 plugins

In reply to Alan Cox

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 9 lines · Alan Cox
Quoted 3 lines · David Masover
Quoted 1 line

What for? It works just fine as it stands, AFAICS.

So does DOS. Do you use DOS? I don't even use DOS to run DOS programs.

False argument. So does the pen, so do hinges on doors. Do you still have hinges on your doors - probably.

Indeed. Because there's nothing better -- not because I "like it the way it is".

Quoted 5 lines
Quoted 1 line

"Ain't broke" is the battle cry of stagnation.

Its also the battle cry of everyone over the age of 20 who also has a real job to do 8)

You caught me. I'm not over 20. But I have a real job, with a company that understands the difference between "ain't broke" and "works well".

Quoted 8 lines
Quoted 4 lines

But, there are some things Reiser does better and faster than ext3, even if you don't count file-as-directory and other toys. There's nothing ext3 does better than Reiser, unless you count the compatibility with random bootloaders and low-level tools.

Certainly compared with reiser3 you've missed a few out including resilience to disk errors (nearly nil on reiser3), and SMP scaling.

Actually, I was talking about reiser4. And Hans corrected me on that...

Although resilience to disk errors isn't a design decision. That's what SMART and new hard drives are for. And if you're stubborn enough to keep the same FS around, there's dm-bbr.

I think Hans (or someone) decided that when hardware stops working, it's not the job of the FS to compensate, it's the job of lower layers, or better, the job of the admin to replace the disk and restore from backups.

Quoted 7 lines
Quoted 3 lines

You know how many I've had thrashed on Reiser4? Two. The first one was with a VERY early alpha/beta, and the second one was when I dropped a laptop and the disk failed.

Entirely or bad blocks ? The latter should have a minimal cost on a well designed fs.

I was able to recover from bad blocks, though of course no Reiser that I know of has had bad block relocation built in... But I got all my files off of it, fortunately.

But the disk did fail, completely, later. Lots of loud clicking.

Quoted 6 lines
Quoted 2 lines

Duplication of effort. With plugins, we can optimize the upper layers of ALL filesystems, regardless of the lower layers, in such a way that

In which case the features belong in the VFS as all those with experience and kernel contributions have been arguing.

No one's arguing that. What we're arguing is that there does seem to be a bit of prejudice when fuck-me-with-a-chainsaw and YOU ARE A FOOL FOR ENABLING THIS all got in, and with at least one of those, there doesn't seem to be any intention of changing it -- all of that, and even with an expressed intention to fix the aesthetical problems with Reiser4 later, we can't get the working version in now.

More infuriatingly, I, at least, have a distinct feeling that once all the issues are fixed, an entirely different crowd of benevolent dictators will come around and say that we can't get in because we change the VFS. At least some people on this list have said things to that effect.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrt7MngHNmZLgCUhAQKdXA/+PDpOzZYTVXgF2n4qiFyrmjFeQ6h0n8i6 c/hXx+QUU0Hw5mjq31+jf2vNpDCKQxcE/HTLdJlRfw8az+xklVOfxzEHf9yV41tv mVKMRJYBhzk2mEvKEDNtnw47SQPBAKo9BtJvl7gOEofiPK/f2K/cy8yMUrow1E9D 02PNT0XX8ysoe86Dqip35+sphczkQN8gilXyUQujNe8edEdkW7PBhbJn92zBQag2 JxA194bquxRyhW78T3tKAEN6/tTPgZYJNy202KC619zzLlK3TslwjjfOQILdRb2i NNkaSQBdYDK70BiFs5Ri7ZbfJHenY6mgGv7yG0vjGF6zjoXtVNsKGrYt9KBL6E1D 392ayxOlWCvBoG9n9sAUzHzcQxmU1lP6OHcO9xWrL6ySD7Fzv4rCtM0uo7gXOzNB aDl5vK7q+ysEjOXZJWT0ikj5ndATCv0Ry8wnt1uL/uktOuaE0egwsouU0jgCDgx1 8Ib8KX/B6bhCy13WkYLPTb6Yg0k+ph0BUyONEcN5cxJIcqfcEt/4Un4MYM+CjGck KLYrrZDclZr8p/paWdNqx1dI9NIBn+F3u78OcWY3NIGfZPh1jmSdG3LRt9DZo5bZ ua2UlAiAWnEDHNLQyMH2zcji31DMqIUwmQ5bX9isBcxt8LORX+IKxoLdUICoY8JS Ii5kNWJAjf4=
=Xsdv
-----END PGP SIGNATURE-----

#152 lore raw

Re: reiser4 plugins

Quoted 2 lines · David Masover
Quoted 1 line · Alan Cox

On Iau, 2005-06-23 at 23:04, David Masover wrote:

[...]

Quoted 9 lines
Quoted 6 lines
Quoted 3 lines
Quoted 1 line

What for? It works just fine as it stands, AFAICS.

So does DOS. Do you use DOS? I don't even use DOS to run DOS programs.

False argument. So does the pen, so do hinges on doors. Do you still have hinges on your doors - probably.

Indeed. Because there's nothing better -- not because I "like it the way it is".

There being nothing better means nobody has ever been able to come up with a better way. Most of the time at least.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

"Ain't broke" is the battle cry of stagnation.

Its also the battle cry of everyone over the age of 20 who also has a real job to do 8)

You caught me. I'm not over 20. But I have a real job, with a company that understands the difference between "ain't broke" and "works well".

"Doesn't work well" /is/ the definition of "broken". Modulo how irritating the "not working well" is...

Quoted 10 lines
Quoted 7 lines
Quoted 4 lines

But, there are some things Reiser does better and faster than ext3, even if you don't count file-as-directory and other toys. There's nothing ext3 does better than Reiser, unless you count the compatibility with random bootloaders and low-level tools.

Certainly compared with reiser3 you've missed a few out including resilience to disk errors (nearly nil on reiser3), and SMP scaling.

Actually, I was talking about reiser4. And Hans corrected me on that...

Quoted 8 lines

Although resilience to disk errors isn't a design decision. That's what SMART and new hard drives are for. And if you're stubborn enough to keep the same FS around, there's dm-bbr.

I think Hans (or someone) decided that when hardware stops working, it's not the job of the FS to compensate, it's the job of lower layers, or better, the job of the admin to replace the disk and restore from backups.

Handling other people's data this way is just reckless irresponsibility. Sure, you can get high performance if you just forego some of your basic responsibilities.

Quoted 10 lines
Quoted 6 lines
Quoted 3 lines

You know how many I've had thrashed on Reiser4? Two. The first one was with a VERY early alpha/beta, and the second one was when I dropped a laptop and the disk failed.

Entirely or bad blocks ? The latter should have a minimal cost on a well designed fs.

I was able to recover from bad blocks, though of course no Reiser that I know of has had bad block relocation built in... But I got all my files off of it, fortunately.

And if the bad blocks had been on files?

[...]

Quoted 10 lines
Quoted 2 lines

In which case the features belong in the VFS as all those with experience and kernel contributions have been arguing.

No one's arguing that. What we're arguing is that there does seem to be a bit of prejudice when fuck-me-with-a-chainsaw and YOU ARE A FOOL FOR ENABLING THIS all got in, and with at least one of those, there doesn't seem to be any intention of changing it -- all of that, and even with an expressed intention to fix the aesthetical problems with Reiser4 later, we can't get the working version in now.

Quoted 5 lines

More infuriatingly, I, at least, have a distinct feeling that once all the issues are fixed, an entirely different crowd of benevolent dictators will come around and say that we can't get in because we change the VFS. At least some people on this list have said things to that effect.

The dictators here have changed over time, sure. Their general attitude has been the same all the way through, since the list started. And you might not like it that way, but I am sure Linux is of the current high quality exactly because any core code that gets into the kernel (and a new filesystem that wants to be central is clearly of this kind) is checked by multiple people with (sometimes widely) diverging backgrounds, interests, and views. Linus himself has stated more than once that his principal job isn't to integrate code into the kernel, but leaving stuff out.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#153 lore raw

Re: reiser4 plugins

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 8 lines · Horst von Brand
Quoted 3 lines · David Masover
Quoted 1 line · Alan Cox

On Iau, 2005-06-23 at 23:04, David Masover wrote:

[...]

[...]

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line

"Ain't broke" is the battle cry of stagnation.

Its also the battle cry of everyone over the age of 20 who also has a real job to do 8)

Quoted 6 lines
Quoted 2 lines

You caught me. I'm not over 20. But I have a real job, with a company that understands the difference between "ain't broke" and "works well".

"Doesn't work well" /is/ the definition of "broken". Modulo how irritating the "not working well" is...

Well, I agree with you there, but to most people, "broken" = "Doesn't work at all." Stagnation happens when people cling to the old way of doing something until they are actually forced to change.

One example of this is energy. Why do we still burn coal and oil? Even lithium ion batteries still give a 200-300 mile range on one electric car I was looking at. There is enough wind in Iowa alone to power half the country, and if we were to put up a bunch of solar panels in Nevada, we'd be done. With the money we would save on not buying oil, we could finish the fuel-cell technologies being worked on, and eventually, the US would get its energy entirely from clean sources.

For that matter, we'd easily save enough money, not to mention the public health benefits (no more smog), to pay for the transition (building hydrogen pumping stations and such, buying new cars...)

But people don't like to change. So we burn oil and make smog and choke on our own stupidity.

That debate really belongs off-list, by the way. All flames on this topic, hit "reply", not "reply-to-all".

Quoted 23 lines
Quoted 17 lines
Quoted 8 lines
Quoted 4 lines

But, there are some things Reiser does better and faster than ext3, even if you don't count file-as-directory and other toys. There's nothing ext3 does better than Reiser, unless you count the compatibility with random bootloaders and low-level tools.

Certainly compared with reiser3 you've missed a few out including resilience to disk errors (nearly nil on reiser3), and SMP scaling.

Actually, I was talking about reiser4. And Hans corrected me on that...

Although resilience to disk errors isn't a design decision. That's what SMART and new hard drives are for. And if you're stubborn enough to keep the same FS around, there's dm-bbr.

Quoted 4 lines

I think Hans (or someone) decided that when hardware stops working, it's not the job of the FS to compensate, it's the job of lower layers, or better, the job of the admin to replace the disk and restore from backups.

Quoted 3 lines

Handling other people's data this way is just reckless irresponsibility. Sure, you can get high performance if you just forego some of your basic responsibilities.

Who's responsibility?

It's the responsibility of the user to back up their files. Sure, this time it was just a little dust, but maybe next time it'll be coffee spilled on the box. Or maybe a power surge. Maybe someone just gets mad enough and kicks the computer very hard.

Reiser4 has been absolutely immune to power loss for me, and quite stable in the face of some corruption, but as soon as I see bad blocks, I throw out the disk, I don't try to make the FS deal with it. And that's all that's missing right now -- a way to use a disk which has some bad blocks -- and that's implemented by dm-bbr if you're that stubborn.

Ultimately, disks are cheap. Cheaper than data loss.

Quoted 9 lines
Quoted 7 lines
Quoted 7 lines
Quoted 3 lines

You know how many I've had thrashed on Reiser4? Two. The first one was with a VERY early alpha/beta, and the second one was when I dropped a laptop and the disk failed.

Entirely or bad blocks ? The latter should have a minimal cost on a well designed fs.

Quoted 6 lines
Quoted 3 lines

I was able to recover from bad blocks, though of course no Reiser that I know of has had bad block relocation built in... But I got all my files off of it, fortunately.

And if the bad blocks had been on files?

Then no FS would save me.

Quoted 5 lines

[...]

Quoted 2 lines
Quoted 2 lines

In which case the features belong in the VFS as all those with experience and kernel contributions have been arguing.

[...]

Quoted 7 lines
Quoted 5 lines

More infuriatingly, I, at least, have a distinct feeling that once all the issues are fixed, an entirely different crowd of benevolent dictators will come around and say that we can't get in because we change the VFS. At least some people on this list have said things to that effect.

Quoted 8 lines

The dictators here have changed over time, sure. Their general attitude has been the same all the way through, since the list started. And you might not like it that way, but I am sure Linux is of the current high quality exactly because any core code that gets into the kernel (and a new filesystem that wants to be central is clearly of this kind) is checked by multiple people with (sometimes widely) diverging backgrounds, interests, and views. Linus himself has stated more than once that his principal job isn't to integrate code into the kernel, but leaving stuff out.

I mean, half of the benevolent dictators at any given time don't seem to like reiser4 for some reason or other. I get the feeling that if we satisfy that half, the way in which we do it will irritate the other half. But I don't know, and I'm going to leave the more specific debates soon.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrzRmHgHNmZLgCUhAQIOyw//bL4sUYQyGSwjKnKZxTznzNnchn0rpNsx PnxrQEvETtkqFlidToJPKDEP3H32tE5W3SeONhIprbQVOK9BagI5Ojm+AhM8pXXa 8K0NJmedlnjMJp5yAxbB+dIwPQ8IN8cG7cdeZCi4ij76yKiDFEqZHXrpm/bvXcbB jzzOF86XPVfexGOUQkwDpR0gmFeY+zIp3+7TGKwG17xHYOej7KpyQctYB7e1me9t MKFll/2USUAglvtdHQ6on2T9slmBTRMny4OFJXKnBTo6rILBTvo6sxC3hG2lv1lN RYXHOd/AV/JQHzhZ1f1vRAn9VxdsUkMgvcOq3k3+ofUP86b1VqxsqGP4Mv3S8uka hghTqBh7ivOr5llCwUHudDIXjRzNw1UBwi6UBC+YB0QQzR3vzc3Q0rvkD1v6hoV+ Li+sX0GhgjaIDzTE+e1rJOAlTAIxeQ9dRMDVBOR5b9HVtkxZ20SSdagDSeHz9JPH Lv0JmqkvYgVf703a3wt56FpTIMd7XvY6dUSR9u3k1xUGQg2jeJA0gZ3tySSogKbT xick+z+m1THTIyA5tXA1JRS1+GNAc4yCL4mHPcUKVWPsKfxaIDB4/+6eCll7Ctxz XzP6s8nv4HkPAsWe1iLvJmuUXdiradC87MJiDGCKM6mbNZ0hFGnKa8zG8nI4C/Ua qHz+cXVkgi8=
=QFCz
-----END PGP SIGNATURE-----

#154 lore raw

Re: reiser4 plugins

In reply to Horst von Brand
Quoted 9 lines · Horst von Brand
Quoted 4 lines · David Masover

I think Hans (or someone) decided that when hardware stops working, it's not the job of the FS to compensate, it's the job of lower layers, or better, the job of the admin to replace the disk and restore from backups.

Handling other people's data this way is just reckless irresponsibility. Sure, you can get high performance if you just forego some of your basic responsibilities.

Your honest-to-bog opinion is that the FS vendor is responsible for the admin not taking backups or the hardware vendor shipping crap?

*still trying to understand how that can be*

Signature · 1 line
mjt
#155 lore raw

Re: reiser4 plugins

Quoted 7 lines · Markus T?rnqvist
Quoted 6 lines · Horst von Brand
Quoted 4 lines · David Masover

I think Hans (or someone) decided that when hardware stops working, it's not the job of the FS to compensate, it's the job of lower layers, or better, the job of the admin to replace the disk and restore from backups.

Quoted 8 lines
Quoted 3 lines

Handling other people's data this way is just reckless irresponsibility. Sure, you can get high performance if you just forego some of your basic responsibilities.

Your honest-to-bog opinion is that the FS vendor is responsible for the admin not taking backups or the hardware vendor shipping crap?

*still trying to understand how that can be*

Most Linux users are using PC-class hardware. And Ted's First Law of PC-Class Hardware is: "Most of it is crap". And then there's Ted's Second Law, "Too many system administrators don't do backups". This is because most system admins are users who've never been trained to be a sysadmin, or who haven't (yet) had weeks or months of works disappear after a hardware failure.

So it's a matter of matching the filesystem to the needs of the user. If you have a filesystem which is blazingly fast, but which at the slightest sign of trouble, trashes your data, versus one which is fast but perhaps not-so-fast as the other filesystem, but which is much more reliable, which would you choose?

XFS has similar issues where it assumes that hardware has powerfail interrupts, and that the OS can use said powerfail interrupt to stop DMA's in its tracks on an power failure, so that you don't have garbage written to key filesystem data structures when the memory starts suffering from the dropping voltage on the power bus faster than the DMA engine or the disk drives. So XFS is a great filesystem --- but you'd better be running it on a UPS, or on a system which has power fail interrupts and an OS that knows what to do. Ext3, because it does physical block journalling, does not suffer from this problem. (Yes, Resierfs uses logical journalling as well, so it suffers from the same problem.)

So perhaps it's not the job of the FS vendor to be responsible for crap hardware or lazy sysadmins that don't do backups. But a system administrator who knows that he doesn't do backups frequently enough, or is running on cheap, crap hardware, would be wise to consider carefully which filesystem he/she wants to use given the systems configuration and his backup habits.

Me, I'll go for the robust filesystem, just on general principles. As a friend from the large-scale enterprise storage world once put it, "Performance is Job 2. Robustness is Job #1." (Of course, if you want to put your fragile filesystem on a multi-million dollar enterprise storage system such as an IBM Shark or an EMC Symmetrix box, I'm sure IBM or EMC will be happy to sell you one. :-)

- Ted

#156 lore raw

Re: reiser4 plugins

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 18 lines · Theodore Ts'o
Quoted 16 lines · Markus T?rnqvist
Quoted 10 lines · Horst von Brand
Quoted 4 lines · David Masover

I think Hans (or someone) decided that when hardware stops working, it's not the job of the FS to compensate, it's the job of lower layers, or better, the job of the admin to replace the disk and restore from backups.

Handling other people's data this way is just reckless irresponsibility. Sure, you can get high performance if you just forego some of your basic responsibilities.

Your honest-to-bog opinion is that the FS vendor is responsible for the admin not taking backups or the hardware vendor shipping crap?

Quoted 6 lines
Quoted 1 line

*still trying to understand how that can be*

Most Linux users are using PC-class hardware. And Ted's First Law of PC-Class Hardware is: "Most of it is crap". And then there's Ted's Second Law, "Too many system administrators don't do backups".

Not our problem anymore. This is like saying that we should run all filesystems in synchronous mode because some users will grab stuff out of the computer without unmounting it -- even more a problem now that we have SATA, which supports hot-swapping.

Too many system administrators don't do backups? Some day the building will burn down and they will wish they had.

Quoted 1 line

So it's a matter of matching the filesystem to the needs of the user.

My needs are a filesystem which doesn't assume I'm an idiot who can't make backups.

Quoted 4 lines

If you have a filesystem which is blazingly fast, but which at the slightest sign of trouble, trashes your data, versus one which is fast but perhaps not-so-fast as the other filesystem, but which is much more reliable, which would you choose?

Hypothetically? I choose the faster one. Failures happen only rarely, and when they happen, there's no telling how small or large they will be, therefore I keep regular backups, and as soon as I see my hardware starts to fail, I grab what I can off of it, merge that with the latest backup, and buy new hardware.

Quoted 8 lines

XFS has similar issues where it assumes that hardware has powerfail interrupts, and that the OS can use said powerfail interrupt to stop DMA's in its tracks on an power failure, so that you don't have garbage written to key filesystem data structures when the memory starts suffering from the dropping voltage on the power bus faster than the DMA engine or the disk drives. So XFS is a great filesystem --- but you'd better be running it on a UPS, or on a system which has power fail interrupts and an OS that knows what to do.

XFS, Reiser3, and Reiser4 all pass the pull-the-plug test. Maybe I just haven't pushed them hard enough?

Quoted 4 lines

Ext3, because
it does physical block journalling, does not suffer from this problem. (Yes, Resierfs uses logical journalling as well, so it suffers from the same problem.)

I don't actually understand the difference here. Are we talking about metadata journalling vs. data journalling?

Quoted 2 lines

So perhaps it's not the job of the FS vendor to be responsible for crap hardware or lazy sysadmins that don't do backups.

Good! Glad we agree!

Quoted 5 lines

But a system
administrator who knows that he doesn't do backups frequently enough, or is running on cheap, crap hardware, would be wise to consider carefully which filesystem he/she wants to use given the systems configuration and his backup habits.

Given a choice between changing filesystems or getting a Streamload account, I choose Streamload. (streamload.com)

Given a choice between reformatting or editing cron, I'd edit cron.

Quoted 6 lines

Me, I'll go for the robust filesystem, just on general principles. As a friend from the large-scale enterprise storage world once put it, "Performance is Job 2. Robustness is Job #1." (Of course, if you want to put your fragile filesystem on a multi-million dollar enterprise storage system such as an IBM Shark or an EMC Symmetrix box, I'm sure IBM or EMC will be happy to sell you one. :-)

Job 1 is done by either that system or by good backups. Don't overkill Job 1 at the expense of Job 2.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQsAY5XgHNmZLgCUhAQJq6g/9G2GEK0R1kujoqn3tjsTIbgC23iu8byYV d6NHSrmmDV8035oo5iCgjSXbaTlIEyOQQqSB78fBC27eHbGctEZJIpSQfHrWlJu6 UJvvaEyl+sR/Mzq8sButKeSUv/T8RMCBKyhdfSgm28lDM/kt9OGZcrf1P1ChUSeK ZnaToh4SAReRhGzq247o+qa2rW5IuIlpKeYGMZhiOB/tXGC3IDMtUOE9IBkn8KVd TtDN4f74PuDJ9VX8K/tZ9DDtKebOJ1wKvIQ/BwyNXt5g36vwj2UwK6GkT2ZJ3bd6 2XLL24LpIBcVU16bk5WPkADRKe0W7S6AseUj7ggjAK0OL8z889vQ3NWJ/rmT/Wx7 OQi58QBwL89OZGSM2qZFJ09CLEH42GXD5jjTrKt4URudYzoIa45VZxD9YYhzeu5e Gxxj4r2BM8bf/AHczCU/tEQwb6hfqxrq+SImWd1uIsozjeiwInHBC56x2qUDjFB/ kiyO1LVmVzrq+6frjPAg6n/i0zuPfF24SeeHX02UOx0jlNLkq2D4Qu1wDB5XlVTK 4xrN83Uy+dHSuDBvnf2tWTqmBju2YG6zmOl1xWKzi3/AVV6mfg1ur0gX0Q6GiZVa 5Nqp618Cs00ZtSk759eAPQRu7Ico0vTHhA0+rTUhApLYMboGaRN5W8OtQu2utHz8 ALX7pD3DrfE=
=wdYo
-----END PGP SIGNATURE-----

#157 lore raw

Re: reiser4 plugins

Quoted 11 lines · David Masover
Quoted 8 lines

XFS has similar issues where it assumes that hardware has powerfail interrupts, and that the OS can use said powerfail interrupt to stop DMA's in its tracks on an power failure, so that you don't have garbage written to key filesystem data structures when the memory starts suffering from the dropping voltage on the power bus faster than the DMA engine or the disk drives. So XFS is a great filesystem --- but you'd better be running it on a UPS, or on a system which has power fail interrupts and an OS that knows what to do.

XFS, Reiser3, and Reiser4 all pass the pull-the-plug test. Maybe I just haven't pushed them hard enough?

Try doing huge amounts of metadata updates while doing the pull-the-plug test. The problem comes when the disk drive is writing into the inode table or other filesystem metadata at the precise moment when power goes down. If the disk is quiescent, you won't see the issue.

XFS is known to have this problem --- the problem was first told to me by an SGI engineer, and they solved this problem in hardware, by adding a power fail interrupt and larger capacitors to the power supply, and then modifying Irix to frantically run around shutting down all outstanding DMA transfer during the grace period provided by the enlarged capacitors. Unfortunately, PC-class hardware don't have power fail interrupts, so....

I have seen this problem reported on ext2 filesystems, so I know it will happen on at least some, and probably most, PC-class hardware. Fortunately switching to ext3 solves the problem, since metadata updates are first written to the journal first, and the recovery replays the full block, which papers over the problem. Unfortunately, a filesystem which uses logical journalling doesn't have a complete copy of the metadata block in the log (only the logical changes are recorded in the journal). This is more compact, and will result in better numbers in artificial benchmarks that emphasize metadata updates (which is not the case in most real-world workloads in my experience). But it the drawback is that filesystems that journal logical updates instead of physical blocks are vulnerable to this particular lossage mode.

Quoted 2 lines

Given a choice between changing filesystems or getting a Streamload account, I choose Streamload. (streamload.com)

*If* you can afford the upload bandwidth to backup over the network, and *if* you don't mind these gems in their legal T's and C's:

Streamload cannot warrant and does not guarantee, and You should not expect, that all of Your private communications and other personal information will never be disclosed in ways not otherwise described in this Privacy Policy.

As-Is and As-Available. Neither Streamload nor any User, or their respective agents, warrants that the service, Private Content or Public Content, or Your access thereto, will be uninterrupted or error free, and Streamload's services and both Public Content and Private Content are provided on an "as is, as available" basis. Streamload has the right to make changes to its services without notice to You. Neither Streamload nor any User warrants that Public Content or Private Content will be free of viruses or similar contamination or destructive features. You expressly agree to assume any and all risk as to the use, quality, performance, accuracy or completeness of any Private Content, Public Content, or Streamload's services.

And of course, while Streamload is pretty cheap for a minimal account, if you have say, a several hundred gigabytes of data which is lost when your filesystem implodes, they have you over a barrel and charge $$$ in order for you to recover it. But if you think it's this works for you, please, go right ahead. Just don't come whining to us afterwards if you get screwed.

- Ted

#158 lore raw

Re: reiser4 plugins

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 1 line · Theodore Ts'o

On Mon, Jun 27, 2005 at 10:19:01AM -0500, David Masover wrote:

[...]

Quoted 11 lines
Quoted 2 lines

Given a choice between changing filesystems or getting a Streamload account, I choose Streamload. (streamload.com)

*If* you can afford the upload bandwidth to backup over the network, and *if* you don't mind these gems in their legal T's and C's:

Streamload cannot warrant and does not guarantee, and You should not expect, that all of Your private communications and other personal information will never be disclosed in ways not otherwise described in this Privacy Policy.

gpg. Was in my upload script to begin with. I keep my key written many times on a single hidden CD. So long as the isofs can be read, at least one of the copies should be usable.

Quoted 7 lines

As-Is and As-Available. Neither Streamload nor any User, or their respective agents, warrants that the service, Private Content or Public Content, or Your access thereto, will be uninterrupted or error free, and Streamload's services and both Public Content and Private Content are provided on an "as is, as available" basis. Streamload has the right to make changes to its services without notice to You. Neither

They don't have any billing information on me. If they charge me for something, I'll cancel my account.

Quoted 6 lines

Streamload nor any User warrants that Public Content or Private Content will be free of viruses or similar contamination or destructive features. You expressly agree to assume any and all risk as to the use, quality, performance, accuracy or completeness of any Private Content, Public Content, or Streamload's services.

They are giving you unlimited storage for free. Even with all of that, it's a damn good deal.

Quoted 6 lines

And of course, while Streamload is pretty cheap for a minimal account, if you have say, a several hundred gigabytes of data which is lost when your filesystem implodes, they have you over a barrel and charge $$$ in order for you to recover it. But if you think it's this works for you, please, go right ahead. Just don't come whining to us afterwards if you get screwed.

I don't have several hundred gigabytes of dynamic data, and neither does the average desktop user. The several hundred gigabytes of static data I have would be backed up on DVDs anyway.

Now, if we're talking about a sysadmin, then of course you need a better backup solution -- no matter what FS you use.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQsBrxngHNmZLgCUhAQLihw//Qz+m971rPW90Fw9We2pqK49pstfsdk4Z n7kimE7kptdcQX0ndnBa1UJ/1bTdIsaGrL+OFTkqLVKtNfitt8DuhdrbXlgm9gvJ k8mwiopXIcfPA7BZ7vxzudttX/35tRfV2ubosagqWZjnpRgBLE3mEdDRCAJb1Z7+ YkynyekwzLPNo4qxWjhazDLqkBvGz4gntQrAwMoOYFlgpQZHlcDAABdRgd1rv+EA paf/ZD90sGm5ZSodwURA8rUKIhRGqv+a5icJQ/PFWl8Cyjtg61w6toWjeK52tE+t 1AezDvjYy7JBItV3BlScgcCze6lKlAEHXDgjUZzKBllh4+h8b46IR6qjvpEk1YeX jOeJpUwIv/YPtRgfGbUBgVyd5wBWqD4+W7a2qn+GIr6sQT176X+fNFzs6RKy+QlZ Ty6yXGejFoydOhbg8zuhZvbEfJKuZCAv+1JkUijFZI6uVZj4Yychc04+qRybg+SQ yY+52KbHyeeH3yQSQrGNa3ngEh84LJjoUpvYXGhQg+ex7kDuQAYlqdUAd7NIQIFj LFvGzkVT/8SLAOi9D7lXB0g52+W4Atl7a3nYpFhIkm2AthNjUkSFhZxyu6wmhFrj 5SPReftDmp4MVQeC4FVZfB72muY4zcO/WYSVmpGn2P89lHEgD4uw8gqQ0XAi5xe6 Gwdb2XcczCo=
=+R+m
-----END PGP SIGNATURE-----

#159 lore raw

Re: reiser4 plugins

Quoted 11 lines · David Masover
Quoted 4 lines

Streamload cannot warrant and does not guarantee, and You should not expect, that all of Your private communications and other personal information will never be disclosed in ways not otherwise described in this Privacy Policy.

gpg. Was in my upload script to begin with. I keep my key written many times on a single hidden CD. So long as the isofs can be read, at least one of the copies should be usable.

They don't have any billing information on me. If they charge me for something, I'll cancel my account.

If you are using the free service, and are encrypting the data, you are explicitly violating their terms of service, and they can delete your data at any time, once they notice.

- Ted

#160 lore raw

Re: reiser4 plugins

Quoted 10 lines · Theodore Ts'o
Quoted 9 lines · David Masover
Quoted 4 lines

Streamload cannot warrant and does not guarantee, and You should not expect, that all of Your private communications and other personal information will never be disclosed in ways not otherwise described in this Privacy Policy.

gpg. Was in my upload script to begin with. I keep my key written many times on a single hidden CD. So long as the isofs can be read, at least one of the copies should be usable.

Quoted 7 lines
Quoted 2 lines

They don't have any billing information on me. If they charge me for something, I'll cancel my account.

If you are using the free service, and are encrypting the data, you are explicitly violating their terms of service, and they can delete your data at any time, once they notice.

Does not look like it:

3c. No encryption and/or steganography for the purpose of circumventing Streamload's rules.
... For example, if you'd like to encrypt something for an extra sense of security and privacy, please feel free to do so. However, when these tools are used solely for the purpose of circumventing Streamload's rules, as determined by the sole discretion of Streamload, the files will be deleted.

Signature · 1 line
Dmitry
#161 lore raw

backup (was Re: reiser4 plugins)

Quoted 24 lines · Dmitry Torokhov
Quoted 11 lines · Theodore Ts'o
Quoted 9 lines · David Masover
Quoted 4 lines

Streamload cannot warrant and does not guarantee, and You should not expect, that all of Your private communications and other personal information will never be disclosed in ways not otherwise described in this Privacy Policy.

gpg. Was in my upload script to begin with. I keep my key written many times on a single hidden CD. So long as the isofs can be read, at least one of the copies should be usable.

Quoted 7 lines
Quoted 2 lines

They don't have any billing information on me. If they charge me for something, I'll cancel my account.

If you are using the free service, and are encrypting the data, you are explicitly violating their terms of service, and they can delete your data at any time, once they notice.

Does not look like it:

Quoted 7 lines

3c. No encryption and/or steganography for the purpose of circumventing Streamload's rules.
... For example, if you'd like to encrypt something for an extra sense of security and privacy, please feel free to do so. However, when these tools are used solely for the purpose of circumventing Streamload's rules, as determined by the sole discretion of Streamload, the files will be deleted.

Trasnlation: They really, honestly, have no way at all of knowing if my GPG-encrypted 450 meg tarball violates their terms. They have legally given themselves the right to delete it just because they think it might violate their terms, but it wouldn't make sense for them to delete a file called "backup.tar.bz2.gpg" when there are no doubt a lot of other encrypted files on their servers. It wouldn't make sense because they would either have to delete such files completely at random, or they'd have to delete all of them at once.

Further translation: This was probably put here to keep the lawyers happy. They don't want to be held liable for illegal stuff on their servers, for example, so they have at least spelled out, in legalese, that illegal stuff (and probably a few other categories) is not allowed on their servers, and that encyrpting it doesn't magically make it legal.

This kind of stuff makes lawyers happy, even if it *practically* does nothing at all.

Regardless, I feel reasonably safe with a backup there. If I ever start to feel unsafe, I can always back up the most critical stuff to gmail, and create a few permanent copies on DVDs. But I feel considerably safer with Streamload than with Gmail, because using Gmail for my own custom backup app directly violates their terms of service.

I think we should officially kill this particular thread.

#162 lore raw

Re: backup (was Re: reiser4 plugins)

Rule #3 from http://www.streamload.com/About/Legal_eng.asp?page=id73#
is pretty clear about what applies if you have a trial account (which seems to be what you have since you say you'll cancel your account if they charge you anything):

3. Do not circumvent Freeloader download restrictions. Any attempt to circumvent Freeloader trial account restrictions will result in a permanent banishment from the Streamload community. This includes (3a) files with an invalid or disguised file format; (3b) encryption; (3c) steganography and/or (3d) creating multiple freeloader accounts.

You can interpret that whatever way you like, but if you're that cavalier with your data, hey, I'm not sure _I'd_ trust your judgement about which filesystem to trust. :-)

- Ted

#163 lore raw

Re: backup (was Re: reiser4 plugins)

Quoted 12 lines · Theodore Ts'o

Rule #3 from http://www.streamload.com/About/Legal_eng.asp?page=id73#
is pretty clear about what applies if you have a trial account (which seems to be what you have since you say you'll cancel your account if they charge you anything):

3. Do not circumvent Freeloader download restrictions. Any attempt to circumvent Freeloader trial account restrictions will result in a permanent banishment from the Streamload community. This includes (3a) files with an invalid or disguised file format; (3b) encryption; (3c) steganography and/or (3d) creating multiple freeloader accounts.

Quoted 3 lines

You can interpret that whatever way you like, but if you're that cavalier with your data, hey, I'm not sure _I'd_ trust your judgement about which filesystem to trust. :-)

Streamload is one possibility. There are others -- gmail and such. This is just what can be had *for*free*.

For that matter, the uploading process doesn't require a user to agree to those terms at all. There's a module in CPAN that allows an upload, and never asks me to agree to any terms, nor does it ask me for an account password.

I don't have multiple accounts, I don't *use* the encryption to attempt to circumvent the download restrictions, especially considering that said download restrictions don't even come into play until I need a restore. By the time I do need a restore, I'm willing to pay money, as 100 megs a month is certainly not going to be enough.

#164 lore raw

Re: reiser4 plugins

In reply to Dmitry Torokhov
Quoted 10 lines · Dmitry Torokhov
Quoted 7 lines

If you are using the free service, and are encrypting the data, you are explicitly violating their terms of service, and they can delete your data at any time, once they notice.

Does not look like it:

Quoted 10 lines

3c. No encryption and/or steganography for the purpose of circumventing Streamload's rules.
... For example, if you'd like to encrypt something for an extra sense of security and privacy, please feel free to do so. However, when these tools are used solely for the purpose of circumventing Streamload's rules, as determined by the sole discretion of Streamload, the files will be deleted.

Oops, sorry, got that wrong, ok, it is more that they provide this service and the terms are at their convenience not anyone else's.

#165 lore raw

Re: reiser4 plugins

In reply to Theodore Ts'o

Wow. Good that you noticed it. It sounds like they have the business model of an intelligence agency.....

Hans

Quoted 7 lines · Theodore Ts'o

If you are using the free service, and are encrypting the data, you are explicitly violating their terms of service, and they can delete your data at any time, once they notice.

- Ted

#166 lore raw

Re: reiser4 plugins

In reply to Theodore Ts'o

Steve, there is a remark about XFS below which you are going to be more expert on.

Quoted 8 lines · Theodore Ts'o

Most Linux users are using PC-class hardware. And Ted's First Law of PC-Class Hardware is: "Most of it is crap". And then there's Ted's Second Law, "Too many system administrators don't do backups". This is because most system admins are users who've never been trained to be a sysadmin, or who haven't (yet) had weeks or months of works disappear after a hardware failure.

The above I agree with. Your words below though are irresponsible.

I get users who tell me that ext* is crap and fsck.ext2 corrupted their filesystem and thats why they use ReiserFS. A difference between us is that I tell them that with all the major linux filesystems (I include XFS and JFS in this) it is by this time far more likely to be hardware that caused corruption than the filesystem software, whereas I guess you tell them something else.

Describing us as trashing data at the slightest sign of trouble is irresponsible. We do block journaling not logical journaling, but there is absolutely nothing inherently wrong with logical journaling, and no reason why it could not be just as robust if that is what XFS uses, so I suspect the rest of your remarks about XFS are unlikely to be sound.

I would say much more, but I am trying to avoid flames this week.

Ted, please try to consider that maybe your competitors also do a fairly good job at the mundane but important aspects of filesystem work, and lets all avoid FUDing each others work. Both ext* and XFS are excellent filesystems, and Linux is lucky to have 3 of the 4 best filesystems available for it.

It will be interesting to see though whether Dominic Giampaolo beats all three of us in the next five years. He probably won't have the performance, but he sure might have the semantic features, and he is very bright. If MS ever makes good use of the talent they have been hiring, we'll have a hard time there also. We may get lucky on that though.....;-)

Quoted 37 lines

So it's a matter of matching the filesystem to the needs of the user. If you have a filesystem which is blazingly fast, but which at the slightest sign of trouble, trashes your data, versus one which is fast but perhaps not-so-fast as the other filesystem, but which is much more reliable, which would you choose?

XFS has similar issues where it assumes that hardware has powerfail interrupts, and that the OS can use said powerfail interrupt to stop DMA's in its tracks on an power failure, so that you don't have garbage written to key filesystem data structures when the memory starts suffering from the dropping voltage on the power bus faster than the DMA engine or the disk drives. So XFS is a great filesystem --- but you'd better be running it on a UPS, or on a system which has power fail interrupts and an OS that knows what to do. Ext3, because it does physical block journalling, does not suffer from this problem. (Yes, Resierfs uses logical journalling as well, so it suffers from the same problem.)

So perhaps it's not the job of the FS vendor to be responsible for crap hardware or lazy sysadmins that don't do backups. But a system administrator who knows that he doesn't do backups frequently enough, or is running on cheap, crap hardware, would be wise to consider carefully which filesystem he/she wants to use given the systems configuration and his backup habits.

Me, I'll go for the robust filesystem, just on general principles. As a friend from the large-scale enterprise storage world once put it, "Performance is Job 2. Robustness is Job #1." (Of course, if you want to put your fragile filesystem on a multi-million dollar enterprise storage system such as an IBM Shark or an EMC Symmetrix box, I'm sure IBM or EMC will be happy to sell you one. :-)

- Ted

#167 lore raw

Re: reiser4 plugins

Quoted 19 lines · Hans Reiser

Steve, there is a remark about XFS below which you are going to be more expert on.

Quoted 13 lines · Theodore Ts'o

XFS has similar issues where it assumes that hardware has powerfail interrupts, and that the OS can use said powerfail interrupt to stop DMA's in its tracks on an power failure, so that you don't have garbage written to key filesystem data structures when the memory starts suffering from the dropping voltage on the power bus faster than the DMA engine or the disk drives. So XFS is a great filesystem --- but you'd better be running it on a UPS, or on a system which has power fail interrupts and an OS that knows what to do. Ext3, because it does physical block journalling, does not suffer from this problem. (Yes, Resierfs uses logical journalling as well, so it suffers from the same problem.)

I presume Ted is referring to problems guaranteeing the integrity of the journal at recovery time. I am coming into this without all the available context, so I may be barking up the wrong tree.... In particular, I am not sure how journaling whole blocks protects you from this.

The xfs journal protects itself against partial writes, to a certain degree. The header of a journal write (inside a 512 byte sector) contains an array of words which are swapped out from the start of each following 512 byte sector of the journal write. The following sectors then each have the log sequence number (LSN) of the write inserted in place of that data.

During recovery, we find the most recent LSN via a binary chop search, this gives us an associated tail LSN. A scan backwards from the head LSN is then done - this covers the total possible amount of in flight data (maximum log buffers x maximum log buffer size). If any of the sectors has the wrong LSN in the first word, then it an all following data is discarded from replay. Of course, we will also not replay any journal entry for which we do not find the transaction commit record.

Now, this protects against some failure cases, it assumes that sector writes are atomic, they either happen or they do not happen. If sector writes are not atomic and one end can be good with the other is bad, then a partial sector is possibly going to get replayed. There have been discussions about doing this with the head and tail of each sector, or using a checksum instead.

XFS on linux has had power cycle crash testing, but there is no way you can cover all possible hardware configurations, and I seem to recall some hardware never recovered from this testing, by that I mean the PC did not survive the continual power cycling and went up in smoke.

Steve

#168 lore raw

Re: reiser4 plugins

Quoted 5 lines · Steve Lord

I presume Ted is referring to problems guaranteeing the integrity of the journal at recovery time. I am coming into this without all the available context, so I may be barking up the wrong tree.... In particular, I am not sure how journaling whole blocks protects you from this.

Actually, I was talking about the problem what happens when power fails while DMA'ing to the disk, and memory, which is more sensitive to voltage drops than the rest of the system, starts sending garbage to the bus, which the disk then faithfully writes to the inode table.

As I recall, you were the one who told me about this problem, and how it was fixed in Irix by using a powerfail interrupt to abort DMA transfers, as well as giving me a program which tests for this condition (basically it writes known test pattern to the disk, and then you do an unclean shutdown, and you look to see if garbage is written to the disk instead of one of the known test patterns). If it wasn't you, it must have been Jim Mostek --- but I could have sworn it was you.....

- Ted

#169 lore raw

Re: reiser4 plugins

Quoted 25 lines · Theodore Ts'o
Quoted 7 lines · Steve Lord

I presume Ted is referring to problems guaranteeing the integrity of the journal at recovery time. I am coming into this without all the available context, so I may be barking up the wrong tree.... In particular, I am not sure how journaling whole blocks protects you from this.

Actually, I was talking about the problem what happens when power fails while DMA'ing to the disk, and memory, which is more sensitive to voltage drops than the rest of the system, starts sending garbage to the bus, which the disk then faithfully writes to the inode table.

As I recall, you were the one who told me about this problem, and how it was fixed in Irix by using a powerfail interrupt to abort DMA transfers, as well as giving me a program which tests for this condition (basically it writes known test pattern to the disk, and then you do an unclean shutdown, and you look to see if garbage is written to the disk instead of one of the known test patterns). If it wasn't you, it must have been Jim Mostek --- but I could have sworn it was you.....

Quoted 5 lines

- Ted

Maybe a more positive approach would be to say, hey, I got this program from the XFS guys that tests for DMA problems on power failure, maybe all the Linux filesystem developers should give it a try?

Shall we try that style of interaction?

Ted, if this corruption hits an ext3 directory structure, and that corrupted directory structure commits, what happens? With ReiserFS (Chris please comment here) it would have to corrupt the directory in the journal, return successful IO, then after that point not corrupt the commit block and have enough power to write the commit block. How is that different for XFS or ext3?

Hans

#170 lore raw

Re: reiser4 plugins

In reply to Theodore Ts'o
Quoted 23 lines · Theodore Ts'o
Quoted 5 lines · Steve Lord

I presume Ted is referring to problems guaranteeing the integrity of the journal at recovery time. I am coming into this without all the available context, so I may be barking up the wrong tree.... In particular, I am not sure how journaling whole blocks protects you from this.

Actually, I was talking about the problem what happens when power fails while DMA'ing to the disk, and memory, which is more sensitive to voltage drops than the rest of the system, starts sending garbage to the bus, which the disk then faithfully writes to the inode table.

As I recall, you were the one who told me about this problem, and how it was fixed in Irix by using a powerfail interrupt to abort DMA transfers, as well as giving me a program which tests for this condition (basically it writes known test pattern to the disk, and then you do an unclean shutdown, and you look to see if garbage is written to the disk instead of one of the known test patterns). If it wasn't you, it must have been Jim Mostek --- but I could have sworn it was you.....

Quoted 2 lines

- Ted

Your memory is better than mine, not sure about the test program, but there was at one point a scenario like that on Irix, and I quite probably did mention it to you. That was certainly not PC hardware, but more likely a large Irix system with multiple power busses spread around a large amount of hardware. I presume you are saying that the fact that ext3 journals complete inode blocks rather that subsections of inodes means that, should a similar trashing of metadata occur during power down, a journal replay would fix it?

I see no way short of hardware fixes of avoiding the general problem of a system failing in an ugly manner like this. Unless you write everything to disk twice (i.e. journal all data), you can still end up with a legitimate set of metadata, and the master copy of your employee database full of nasty little bits of corruption.

Steve

#171 lore raw

Re: reiser4 plugins

Quoted 5 lines · Steve Lord

I see no way short of hardware fixes of avoiding the general problem of a system failing in an ugly manner like this. Unless you write everything to disk twice (i.e. journal all data), you can still end up with a legitimate set of metadata, and the master copy of your employee database full of nasty little bits of corruption.

If I as a user may add my own little experience with different fs on my computer, where I tend to use kernels or features that cause my machine to (hard) lock up:

2.4.2x kernel and reiserfs (V3): This was the only one which really managed to smoke my partition...

2.6.x-test-y till 2.6.x stable kernel and reiserfs (V3): I did a short intermezzo with xfs but at that time it felt too slow to bear, so I gave reiserfs another chance. And here it was very robust. No lock-up did serious damage to the data. Only currently accessed files were partly overwritten with @^@^@^ and so on...

After a year or so of heavy usage the formerly fast reiserfs partition became more and more slow till I got sick of it. So I tried JFS and forced a hard lock. Data was OK, but (probably error in gentoo scripts) fsck wasn't automatically called (or failed, I don't remember) and I had to do it by hand, which was a major annoyance - I thought then.

So I gave ext3 a try. Very robust, but at the same time slooow. I couldn't bear it after some months. So I gave xfs another try. Yes, now it felt much better. Still not that fast as reiserfs, IIRC, but better than the first time I tried. I am still having xfs on / and it works pretty well, and is rather robust against hard locks with about the same amount of data losing as reiserfs. But what annoys me very much, is that I have to run xfs_repair by hand and by booting from another partition. Even after a hard lock, the partition mounts w/o problems and everything seems OK, but it only seems like that. In fact after some hours/days of use, you'll notice oddities, like files or directories which cannot be removed and things like that. After running xfs_repair everything is back in order.

In between I gave an alpha (or rather several alphas) of reiser4 a try - but not on /, just on /usr. Well, I wouldn't say it was extraordinary fast. In fact it felt slower than reiserfs V3, but much more space efficient. And to my surprise it was very robust as well. Hard-locks were no problem. Only annoying then was, that unmounting regularly produced oops but later versions corrected that. But nevertheless it didn't survive, as like V3, with time V4 became slower and slower. In this case no year was needed, but just one month or alike. So end of test...but in fact I'll give V4 another go in the near future.

All in all I can say that every fs I tested was able to not smoke all of my data, even using an instable machine - but only ext3, reiser v3 and v4 were non-annoying. But xfs is majorly annoying in that respect. I hope that new versions will be able to keep consistency w/o having to run xfs_repair every time after a lock-up...

But what I don't understand is, that sometimes even files, which were only opened for reading, got overwritten with @^@^@ after a lock-up. I don't understand the logics here, how this could happen.

Thx for your time,

Prakash

#172 lore raw

Re: reiser4 plugins

Quoted 6 lines · Prakash Punnoor

. But nevertheless it didn't survive, as like V3, with time V4 became slower and slower. In this case no year was needed, but just one month or alike. So end of test...but in fact I'll give V4 another go in the near future.

Interesting that it got slower with time. It sounds like our online repacker is much needed. It will be a priority for after the kernel merge.

#173 lore raw

Re: reiser4 plugins

Quoted 10 lines · Hans Reiser
Quoted 6 lines · Prakash Punnoor

. But nevertheless it didn't survive, as like V3, with time V4 became slower and slower. In this case no year was needed, but just one month or alike. So end of test...but in fact I'll give V4 another go in the near future.

Interesting that it got slower with time. It sounds like our online repacker is much needed. It will be a priority for after the kernel merge.

I told y'all that last year on the reiserfs-list, with benchmarks :-)

Signature · 1 line
Zan Lynx <[email protected]>
#174 lore raw

Re: reiser4 plugins

In reply to Hans Reiser
Quoted 5 lines · Hans Reiser
Quoted 3 lines

. But nevertheless it didn't survive, as like V3, with time V4 became slower and slower. In this case no year was needed, but just one month or alike. So end of test...but in fact I'll give V4 another go in the near future.

Interesting that it got slower with time. It sounds like our online repacker is much needed. It will be a priority for after the kernel merge.

Whered it go, I recall it being activated with:
echo 1 >/sys/fs/reiser4/*/repacker/start

#175 lore raw

Re: reiser4 plugins

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 6 lines · Gregory Maxwell
Quoted 4 lines · Hans Reiser
Quoted 3 lines

. But nevertheless it didn't survive, as like V3, with time V4 became slower and slower. In this case no year was needed, but just one month or alike. So end of test...but in fact I'll give V4 another go in the near future.

Quoted 6 lines
Quoted 2 lines

Interesting that it got slower with time. It sounds like our online repacker is much needed. It will be a priority for after the kernel merge.

Whered it go, I recall it being activated with:
echo 1 >/sys/fs/reiser4/*/repacker/start

That proved to be unstable (on my box, the repacker never finished) and there was also some talk of making it only available commercially -- that is, pay some amount based on how much storage you're using it on, get the repacker.

I like open source better, but whatever amount was proposed is more than fair for me, especially considering how fast stuff is even without the repacker.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQsGv0XgHNmZLgCUhAQKluQ/+Pjg6h5qUpO/6pG7v8ku2OhyUoR6aneTb FlqjaqNuxqKaT/C3iqZYLZ+YUk+MYdXKUYNthdlY9ZtaPNwnp34/AHUSLtU4Imgw Z4kIn3csKCkpeLRQMJ1o5nRSBwa+38Ta17ktYhn7L+ln53/ElULbPrmxjoSmA+SK ocjkVcUI8azJJosD1Y0tb5BSVa9PC+buPOgbkinvFVeme2ouwG+tAeAMECShface 0FbT1+Ai//S/t5rz/oz6BzlV+K7Jl+8OopFs9MnwD8Gqh21zeGiawLQGsDwvZuEt wJq73Y6b573RVJUy4dHAUqBZen0gar1GPWS2RpB3RwIYgb7/Ui3Rf5t//vTrZMNo Ej9R3AXcU82M8YHHmwr1QJfhrt7kddXPw5TSpA66RWNJdeFxoaXxja6sFFjQbSTs G5BHy2P/oHO8+uhbS+NI8T1N8KsrRrhJWr0pcjhHaoDogGdUj7TZtB+fANgLHbdV GstRVz8QvKGOZ8FrX/mMPzsfUbRx3pq+C/FJzSgqVI/4oTc/XIOS7cb92EC+1WrO KhYI3vy1zeKximboJXrPlr7QXkakjoOSFNH/MVzmF3sP1pe92hey7SXrpfgrOCOa 8gVo0u1tE98fuatl3MA8t3OGXGM5JXKpHVk/FiQt4358wAMmyDizvOeZ3GL1CA/M ixjK/caPrP0=
=LFCF
-----END PGP SIGNATURE-----

#176 lore raw

Re: reiser4 plugins

In reply to Prakash Punnoor
Quoted 12 lines · Prakash Punnoor

So I gave ext3 a try. Very robust, but at the same time slooow. I couldn't bear it after some months. So I gave xfs another try. Yes, now it felt much better. Still not that fast as reiserfs, IIRC, but better than the first time I tried. I am still having xfs on / and it works pretty well, and is rather robust against hard locks with about the same amount of data losing as reiserfs. But what annoys me very much, is that I have to run xfs_repair by hand and by booting from another partition. Even after a hard lock, the partition mounts w/o problems and everything seems OK, but it only seems like that. In fact after some hours/days of use, you'll notice oddities, like files or directories which cannot be removed and things like that. After running xfs_repair everything is back in order.

I don't know what was going on with your systems, but I've been using XFS since the original 1.0 Linux release from SGI and I'd guess that I've had to run xfs_repair less than 10 times and most of them were on Alpha and Sparc64 before issues with those arches got ironed out.

Right now I have XFS on both Alpha and Sparc64 and I haven't had any issues on either box. Infact the Sparc64 box's XFS filesystem was converted from reiser3 because the some part of the filesystem got mysteriously corrupted in such a way that reiserfsck and the reiser3 driver both thought it was fine but accessing a certain file would cause a lockup.

I really hope the reiser4 userland tools are a lot better than the reiser3 tools, that's an area that reiserfs has lagged behind hugely IMO.

Quoted 17 lines

In between I gave an alpha (or rather several alphas) of reiser4 a try - but not on /, just on /usr. Well, I wouldn't say it was extraordinary fast. In fact it felt slower than reiserfs V3, but much more space efficient. And to my surprise it was very robust as well. Hard-locks were no problem. Only annoying then was, that unmounting regularly produced oops but later versions corrected that. But nevertheless it didn't survive, as like V3, with time V4 became slower and slower. In this case no year was needed, but just one month or alike. So end of test...but in fact I'll give V4 another go in the near future.

All in all I can say that every fs I tested was able to not smoke all of my data, even using an instable machine - but only ext3, reiser v3 and v4 were non-annoying. But xfs is majorly annoying in that respect. I hope that new versions will be able to keep consistency w/o having to run xfs_repair every time after a lock-up...

Quoted 7 lines

But what I don't understand is, that sometimes even files, which were only opened for reading, got overwritten with @^@^@ after a lock-up. I don't understand the logics here, how this could happen.

Thx for your time,

Prakash

Jim.

#177 lore raw

Re: reiser4 plugins

Quoted 15 lines · Jim Crilly
Quoted 11 lines · Prakash Punnoor

So I gave ext3 a try. Very robust, but at the same time slooow. I couldn't bear it after some months. So I gave xfs another try. Yes, now it felt much better. Still not that fast as reiserfs, IIRC, but better than the first time I tried. I am still having xfs on / and it works pretty well, and is rather robust against hard locks with about the same amount of data losing as reiserfs. But what annoys me very much, is that I have to run xfs_repair by hand and by booting from another partition. Even after a hard lock, the partition mounts w/o problems and everything seems OK, but it only seems like that. In fact after some hours/days of use, you'll notice oddities, like files or directories which cannot be removed and things like that. After running xfs_repair everything is back in order.

Quoted 4 lines

I don't know what was going on with your systems, but I've been using XFS since the original 1.0 Linux release from SGI and I'd guess that I've had to run xfs_repair less than 10 times and most of them were on Alpha and Sparc64 before issues with those arches got ironed out.

Perhaps it is due to the fact that I use xfs on software RAID-0 and both HDs have 8MB cache write-back enabled? So, all in all 16MB needs to be commited on/before lock-up, maybe too much for xfs? (This situation was no prob for ext3, though. Thinking again, I never used reiser V3 or V4 on the RAID-0, so my comparison might not have been fair.)

Prakash

#178 lore raw

Re: reiser4 plugins

Quoted 21 lines · Prakash Punnoor
Quoted 15 lines · Jim Crilly
Quoted 11 lines · Prakash Punnoor

So I gave ext3 a try. Very robust, but at the same time slooow. I couldn't bear it after some months. So I gave xfs another try. Yes, now it felt much better. Still not that fast as reiserfs, IIRC, but better than the first time I tried. I am still having xfs on / and it works pretty well, and is rather robust against hard locks with about the same amount of data losing as reiserfs. But what annoys me very much, is that I have to run xfs_repair by hand and by booting from another partition. Even after a hard lock, the partition mounts w/o problems and everything seems OK, but it only seems like that. In fact after some hours/days of use, you'll notice oddities, like files or directories which cannot be removed and things like that. After running xfs_repair everything is back in order.

Quoted 4 lines

I don't know what was going on with your systems, but I've been using XFS since the original 1.0 Linux release from SGI and I'd guess that I've had to run xfs_repair less than 10 times and most of them were on Alpha and Sparc64 before issues with those arches got ironed out.

Quoted 5 lines

Perhaps it is due to the fact that I use xfs on software RAID-0 and both HDs have 8MB cache write-back enabled? So, all in all 16MB needs to be commited on/before lock-up, maybe too much for xfs? (This situation was no prob for ext3, though. Thinking again, I never used reiser V3 or V4 on the RAID-0, so my comparison might not have been fair.)

Maybe, I've never used XFS on software RAID-0. The Sparc64 I mentioned has it's XFS filesystem on software RAID-1 and that's introduced no problems, but I've seen way too many drives die to risk running RAID-0.

Quoted 2 lines

Prakash

Jim.

#179 lore raw

RE: reiser4 plugins

In reply to Steve Lord
Quoted 18 lines · Hans Reiser

Steve, there is a remark about XFS below which you are going to be more expert on.

Quoted 13 lines · Theodore Ts'o

XFS has similar issues where it assumes that hardware has powerfail interrupts, and that the OS can use said powerfail interrupt to stop DMA's in its tracks on an power failure, so that you don't have garbage written to key filesystem data structures when the memory starts suffering from the dropping voltage on the power bus faster than the DMA engine or the disk drives. So XFS is a great filesystem --- but you'd better be running it on a UPS, or on a system which has power fail interrupts and an OS that knows what to do. Ext3, because it does physical block journalling, does not suffer from this problem. (Yes, Resierfs uses logical journalling as well, so it suffers from the same problem.)

True now, not so around 2.4.20 when XFS was rock-solid. I think they tried to improve on performance and broke something. I wish they would fix that because it forced me back to ext3, as in consistency over performance any time.

#180 lore raw

Re: reiser4 plugins

In reply to Hans Reiser
Quoted 5 lines · Hans Reiser

A difference between us is
that I tell them that with all the major linux filesystems (I include XFS and JFS in this) it is by this time far more likely to be hardware that caused corruption than the filesystem software, whereas I guess you tell them something else.

Oh, I agree with this, and I do tell people that. The question though is how the filesystem recovers from said hardware-caused corruption once it does happen. You've admitted that reiserfs3 has less than optimal recovery characteristics from hardware-induced corruptions if said filesystem contains reiserfs filesystem images; that would be an example of a filesystem not being as robust as it could be. (It'll be interesting to see if SuSE will support reiserfsv3 in combination with the Xen hypervisor or other virtualization systems, which makes use of filesystem images.) Another example would be DMA'ing garbage into the hard drive after a power failure --- how does a filesystem respond to this eventuality?

You probably hear more stories people who got unlucky with hardware-induced corruptions with ext2/3, and I probably hear more from users who have sworn off of reiserfs just because are sample sets are somewhat biased. Such are the dangers of relying on anecdotal evidence.

However, logically speaking, if a filesystem is designed such that in certain cases, the fsck program has to brute-force search every single disk block looking for data structures that _look_ like they might be part of the filesystem data, well, that's always going to be more error prone than one where the filesystem metadata is in easily-predicted locations. It sounds like you've added some more checks in reiser4, and that's definitely a good thing. Time will tell whether they are sufficient or not.

Regards,

- Ted

#181 lore raw

Re: reiser4 merging action list

Andrew asked me to put together a list of things that need to be done before merging:

* VFS will dispatch directly to the method of the plugin for the
*_operations methods. This requires duplicating to all plugin methods the common code currently used by all reiser4 plugins for a given method. It has the desirable side effect of making the methods more fully self-contained, which is somethng I had wanted two years ago and was a little sad to not get, and the cost of duplicating some code. Since not all plugin methods are *_operations, it means we have two structures with duplicated data, and duplicate data that must be in sync at all times is classical badness in programming technique (see Codd and
normalization). vs owns this task

* review all sparse complaints, and revise as appropriate.

* panic and code beauty: everyone agrees that having function, file,
and line added to reiser4_panic output hurts nothing (I hope). Everyone agrees that restarting the machine without an error message seems like a
useless option to allow. Much else was argued, not sure if anything
was a consensus view. Various detail improvements were suggested by Pecca, and I agreed with half of them.

* metafiles should be disabled until we can present code that works
right. Half the list thinks we cannot solve the cycles problem ever. Disable metafiles and postpone problem until working code, or the failure to produce it, makes it possible to do more than rant at each other. This is currently already done in the -mm patches, but is mentioned lest someone think it forgotten.

* update the locking documentation

Probably I forget something.

Best,

Hans

#182 lore raw

Re: reiser4 merging action list

Quoted 4 lines · Hans Reiser

Andrew asked me to put together a list of things that need to be done before merging:

Thanks.

As I said to Hans, if we can get a list of bullet-point actions nailed down and agreed to then we have an uncontroversial path to happiness and a merge. Let's get down and concentrate on technical specifics.

Hans, please maintain this list and republish it as we work through things.

Quoted 21 lines

* VFS will dispatch directly to the method of the plugin for the
*_operations methods. This requires duplicating to all plugin methods the common code currently used by all reiser4 plugins for a given method. It has the desirable side effect of making the methods more fully self-contained, which is somethng I had wanted two years ago and was a little sad to not get, and the cost of duplicating some code. Since not all plugin methods are *_operations, it means we have two structures with duplicated data, and duplicate data that must be in sync at all times is classical badness in programming technique (see Codd and
normalization). vs owns this task

* review all sparse complaints, and revise as appropriate.

* panic and code beauty: everyone agrees that having function, file,
and line added to reiser4_panic output hurts nothing (I hope). Everyone agrees that restarting the machine without an error message seems like a
useless option to allow. Much else was argued, not sure if anything
was a consensus view. Various detail improvements were suggested by Pecca, and I agreed with half of them.

Quoted 9 lines

* metafiles should be disabled until we can present code that works
right. Half the list thinks we cannot solve the cycles problem ever. Disable metafiles and postpone problem until working code, or the failure to produce it, makes it possible to do more than rant at each other. This is currently already done in the -mm patches, but is mentioned lest someone think it forgotten.

* update the locking documentation

There's also the custom list, hash and debug code. We should either

a) remove them or

b) generify them and submit as standalone works or

c) justify them as custom-to-reiser4 and leave them as-is.

#183 lore raw

Re: reiser4 merging action list

Quoted 51 lines · Andrew Morton
Quoted 4 lines · Hans Reiser

Andrew asked me to put together a list of things that need to be done before merging:

Thanks.

As I said to Hans, if we can get a list of bullet-point actions nailed down and agreed to then we have an uncontroversial path to happiness and a merge. Let's get down and concentrate on technical specifics.

Hans, please maintain this list and republish it as we work through things.

Quoted 21 lines

* VFS will dispatch directly to the method of the plugin for the
*_operations methods. This requires duplicating to all plugin methods the common code currently used by all reiser4 plugins for a given method. It has the desirable side effect of making the methods more fully self-contained, which is somethng I had wanted two years ago and was a little sad to not get, and the cost of duplicating some code. Since not all plugin methods are *_operations, it means we have two structures with duplicated data, and duplicate data that must be in sync at all times is classical badness in programming technique (see Codd and
normalization). vs owns this task

* review all sparse complaints, and revise as appropriate.

* panic and code beauty: everyone agrees that having function, file,
and line added to reiser4_panic output hurts nothing (I hope). Everyone agrees that restarting the machine without an error message seems like a
useless option to allow. Much else was argued, not sure if anything
was a consensus view. Various detail improvements were suggested by Pecca, and I agreed with half of them.

Quoted 11 lines

* metafiles should be disabled until we can present code that works
right. Half the list thinks we cannot solve the cycles problem ever. Disable metafiles and postpone problem until working code, or the failure to produce it, makes it possible to do more than rant at each other. This is currently already done in the -mm patches, but is mentioned lest someone think it forgotten.

* update the locking documentation

Quoted 13 lines

There's also the custom list, hash and debug code. We should either

a) remove them or

b) generify them and submit as standalone works or

c) justify them as custom-to-reiser4 and leave them as-is.

either b) or c) is ok with me for the list code. The debug code should be c) I think.

Probably vs can offer a more detailed and accurate opinion,

Hans

#184 lore raw

Re: reiser4 merging action list

Quoted 7 lines · Andrew Morton
Quoted 7 lines

There's also the custom list, hash and debug code. We should either

a) remove them or

b) generify them and submit as standalone works or

c) justify them as custom-to-reiser4 and leave them as-is.

Quoted 4 lines · Hans Reiser

either b) or c) is ok with me for the list code. The debug code should be c) I think.

Probably vs can offer a more detailed and accurate opinion,

I completely agree that the current state of the generic hashing facilities is somewhat poor but I fail to see why you can't use <linux/list.h>.

As for the debugging code, I would love to see that turned into something generic (every subsystem has their own now) but it is definitely not something that should stop you from merging.

Pekka

#185 lore raw

Re: reiser4 merging action list

Quoted 19 lines · Pekka Enberg
Quoted 9 lines · Andrew Morton
Quoted 9 lines

There's also the custom list, hash and debug code. We should either

a) remove them or

b) generify them and submit as standalone works or

c) justify them as custom-to-reiser4 and leave them as-is.

Quoted 3 lines · Hans Reiser

either b) or c) is ok with me for the list code. The debug code should be c) I think.

Quoted 9 lines
Quoted 3 lines

Probably vs can offer a more detailed and accurate opinion,

I completely agree that the current state of the generic hashing facilities is somewhat poor but I fail to see why you can't use <linux/list.h>.

I'll let vs and maybe nikita comment.

Quoted 9 lines

As for the debugging code, I would love to see that turned into something generic (every subsystem has their own now) but it is definitely not something that should stop you from merging.

Pekka

If I encourage you to make a patch, is that ok of me?

#186 lore raw

Re: reiser4 merging action list

In reply to Hans Reiser
Quoted 3 lines · Hans Reiser

Andrew asked me to put together a list of things that need to be done before merging:

...

Quoted 1 line

Probably I forget something.

I've started to do a very basic look over the tree and there's a few more things that spring to mind:

- cpp abuse. There's quite a lot of really odd macros - the typesafe_list, typesafe_hash stuff is mentioned already, but there's really horrible stuff like _INIT_ and _DONE_ bits in init_super.c, and the wrappers for plugin method invocations.
- endianess handling. The d* types are a lovely attempt to make sure you're not missing endianess conversions. We have a more general way to ensure that now using sparse, that's the __le* / __be* types. Try running sparse -Wbitwise to find places you'll need that annotations (after the normal sparse warnings are fixed, else you'll have a hard time seeing them I guess) - after that the single element struct thing can go away. Also you're defining a CPU_IN_DISK_ORDER macro that's never used..
- lease use kthread_* instead of kernel_thread() + lots of opencoding

#187 lore raw

Re: reiser4 merging action list

In reply to Hans Reiser
Quoted 5 lines · Hans Reiser

Andrew asked me to put together a list of things that need to be done before merging:
...
Probably I forget something.

* remove the dependency on !4KSTACKS

Quoted 3 lines

Best,

Hans

cu
Adrian

Signature · 4 lines
       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed
#188 lore raw

Re: reiser4 plugins

In reply to Theodore Ts'o

What happens when the ext3 inode tables get hit by sector damage? Like us, ext3 has data munging if you hit the wrong thing, its just that our sources of data munging are different in the details. Details matter though, and so we are improving with each release, and V4 does have some real improvements. I hope the next major release will have atomic transactions that don't have that niggling 5% left undone that prevents them from being fully there, and then reiser4 can seriously advance the field of reliability in filesystems rather than just playing catchup.

Good luck with ext3,

Hans

Quoted 16 lines · Theodore Ts'o

However, logically speaking, if a filesystem is designed such that in certain cases, the fsck program has to brute-force search every single disk block looking for data structures that _look_ like they might be part of the filesystem data, well, that's always going to be more error prone than one where the filesystem metadata is in easily-predicted locations. It sounds like you've added some more checks in reiser4, and that's definitely a good thing. Time will tell whether they are sufficient or not.

Regards,

- Ted

#189 lore raw

Re: reiser4 plugins

In reply to Markus Törnqvist
Quoted 7 lines · Markus Törnqvist
Quoted 5 lines · Horst von Brand
Quoted 4 lines · David Masover

I think Hans (or someone) decided that when hardware stops working, it's not the job of the FS to compensate, it's the job of lower layers, or better, the job of the admin to replace the disk and restore from backups.

Quoted 6 lines
Quoted 3 lines

Handling other people's data this way is just reckless irresponsibility. Sure, you can get high performance if you just forego some of your basic responsibilities.

Your honest-to-bog opinion is that the FS vendor is responsible for the admin not taking backups or the hardware vendor shipping crap?

No. But just relying on perfect hardware and concientious sysadmins is reckless. Hardware /is/ flaky, sysadmins /are/ (sometimes) lazy (and besides, today they are increasingly just plain Joe Sixpack users). Also, backing up a few hundred GiB is /not/ fun, and then keeping track of all the backups is messy.

Also, I'm not claiming that they are /solely/ responsible, but not having the filesystem fall apart utterly every time some bug breaths on it /is/ a requirement.

Quoted 1 line

*still trying to understand how that can be*

You haven't been around too much yet, do you?

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#190 lore raw

Re: reiser4 plugins

Quoted 6 lines · Horst von Brand

No. But just relying on perfect hardware and concientious sysadmins is reckless. Hardware /is/ flaky, sysadmins /are/ (sometimes) lazy (and besides, today they are increasingly just plain Joe Sixpack users). Also, backing up a few hundred GiB is /not/ fun, and then keeping track of all the backups is messy.

Even home users have started to set up raid mirrors at home now that disk space is cheap. That's a step in the right direction, I suppose, with hardware never being good.

Taking backups in an environment where you need a few hundred GiB backed up is not that difficult.

Get a separate, redundant box with a big tape changer and drop periodical backups off at your bank's vault.

Get a separate, very reduntant box, with a truckload of proven drives in a separate raid box and run your stuff there.

Get both of the above.

If Joe Sixpack loses his mp3 collection, I don't really care, nor should anyone else. Anything important enough to care about is easy enough to back up. Always.

Arrogance? Maybe.

Quoted 3 lines

Also, I'm not claiming that they are /solely/ responsible, but not having the filesystem fall apart utterly every time some bug breaths on it /is/ a requirement.

Reiserfs does not fall apart utterly every time some bug breaths on it.

Quoted 2 lines
Quoted 1 line

*still trying to understand how that can be*

You haven't been around too much yet, do you?

Rather I take backups, buy better hardware and understand there's a risk involved.

Computers as a complete set can't be trusted, you can only make the best accomodations you can.

Signature · 1 line
mjt
#191 lore raw

Re: reiser4 plugins

Quoted 11 lines · Markus Törnqvist
Quoted 6 lines · Horst von Brand

No. But just relying on perfect hardware and concientious sysadmins is reckless. Hardware /is/ flaky, sysadmins /are/ (sometimes) lazy (and besides, today they are increasingly just plain Joe Sixpack users). Also, backing up a few hundred GiB is /not/ fun, and then keeping track of all the backups is messy.

Even home users have started to set up raid mirrors at home now that disk space is cheap. That's a step in the right direction, I suppose, with hardware never being good.

I've lost more data to my own recklessness and stupidity than filesystem corruption and hardware failure combined. RAID isn't really a good solution. My policy for cheap storage (currently the only variety I own) is when I buy a new hard drive, use the old drive for backups. The old drive is always large enough to hold everything I'd miss and then some. Home users should be capable of doing the same, assuming Windows has something similar to rsync.

Quoted 33 lines

Taking backups in an environment where you need a few hundred GiB backed up is not that difficult.

Get a separate, redundant box with a big tape changer and drop periodical backups off at your bank's vault.

Get a separate, very reduntant box, with a truckload of proven drives in a separate raid box and run your stuff there.

Get both of the above.

If Joe Sixpack loses his mp3 collection, I don't really care, nor should anyone else. Anything important enough to care about is easy enough to back up. Always.

Arrogance? Maybe.

Quoted 3 lines

Also, I'm not claiming that they are /solely/ responsible, but not having the filesystem fall apart utterly every time some bug breaths on it /is/ a requirement.

Reiserfs does not fall apart utterly every time some bug breaths on it.

Quoted 2 lines
Quoted 1 line

*still trying to understand how that can be*

You haven't been around too much yet, do you?

Rather I take backups, buy better hardware and understand there's a risk involved.

Computers as a complete set can't be trusted, you can only make the best accomodations you can.

Signature · 1 line
Jake Maciejewski <[email protected]>
#192 lore raw

Re: reiser4 plugins

In reply to David Masover
Quoted 5 lines · David Masover
Quoted 2 lines

False argument. So does the pen, so do hinges on doors. Do you still have hinges on your doors - probably.

Indeed. Because there's nothing better -- not because I "like it the way it is".

I chose hinges carefully - there are better technologies than the generic hinge and there have been for many many years (plus cool stuff like magnetic doors). You of course aren't a door geek (nor am I I'd point out) but a computing one...

Quoted 3 lines

Although resilience to disk errors isn't a design decision. That's what SMART and new hard drives are for. And if you're stubborn enough to keep the same FS around, there's dm-bbr.

Its very much a file system design consideration. If you get a small amount of corruption or lost blocks which is the usual drive failure case you want a high probability of getting the data back. BSD style bitmap/inode table file systems based on FFS concepts (UFS, ext2, ext3 etc) are very good in that respect and your loss is usually minimal.

Quoted 3 lines

I think Hans (or someone) decided that when hardware stops working, it's not the job of the FS to compensate, it's the job of lower layers, or better, the job of the admin to replace the disk and restore from backups.

Most desktop users today don't have backups because there is no credible backup technology for 500Gb of data. They may have partial backups. Some things the fs can't deal with - if the disk goes boom then thats a lower level concern. Also certain bits like writing to spare blocks on a problem write are indeed handled drive level nowdays.

Quoted 6 lines
Quoted 2 lines

Entirely or bad blocks ? The latter should have a minimal cost on a well designed fs.

I was able to recover from bad blocks, though of course no Reiser that I know of has had bad block relocation built in... But I got all my files off of it, fortunately.

Thats a good sign. reiser3 was very fragile when blocks of disk went for a walk. If you got most of your data back thats a positive sign, not statistically a valid sample but a positive sign

Quoted 4 lines

the issues are fixed, an entirely different crowd of benevolent dictators will come around and say that we can't get in because we change the VFS. At least some people on this list have said things to that effect.

There are four important issues I see here

1. It must work
2. It must be clean code that follows the kernel style
3. It must not break other stuff
4. It needs a maintainer who won't get bored 12 months later and only support reiser5

#3 is the VFS stuff and getting the VFS locking wrong or unclean is a *very* big deal because you'll cause corruption to non reiser4 users.

Alan

#193 lore raw

Re: reiser4 plugins

Quoted 5 lines · Alan Cox

Thats a good sign. reiser3 was very fragile when blocks of disk went for a walk. If you got most of your data back thats a positive sign, not statistically a valid sample but a positive sign

Alan, this is FUD. Our V3 fsck was written after everything else was,
for lack of staffing reasons (why write an fsck before you have an FS worth using). As a result, there was a long period where the fsck code was unstable. It is reliable now.

People often think that our tree makes fsck less robust. Actually fsck can throw the entire internal tree away and rebuild from leaf nodes, and frankly that makes things pretty robust.

There is an area where we suffered from writing fsck last. When there are two leaf nodes with the same key range AND the bitmap cannot be trusted to tell us which is the valid one, we don't know which is the most recent, and pick arbitrarily. Also, if you store a backup of V3, and you don't compress it, and you wipe out the bitmap blocks and need to use fsck, we don't know what blocks are backup image and what blocks are the fs. We advise users to never store a V3 backup on V3 without compressing it. Additionally, if you create a filesystem, use it a lot, and then mkfs on top of it, and use the new filesystem, and then wipe out the bitmap blocks so that we cannot tell what was in the original fs and what is in the current fs, things get merged together. Also, anytime you wipe out the bitmap blocks, you can get files that were deleted. This turns out to be a feature for many users who delete a file, regret it, email us, we usually get it back just fine by ignoring the bitmap blocks and recovering everything we can find.

In V4 we fixed all this. In V4 we started by asking Vitaly our fsck guy what node format changes would make his code easier and better. We put timestamps in every formatted node, and then because we don't trust clocks we also put an mkfsid. Thus, we can distinguish the current and the previously made filesystem even if the clock is moved backwards. We can distinguish backups of other filesystems that were not compressed from the real filesystem. Timestamps allows fixing everything that I described above.

What is more important, to fix V3 as described above would require a disk format change that makes it prohibitive to roll out in a minor release, and in V4, to change the disk format just requires adding a plugin which could be done in a minor release. Not only did we fix the problem in V4, we fixed the meta problem so that any future disk format problems are now no biggie to fix.

Quoted 4 lines

4. It needs a maintainer who won't get bored 12 months later and only support reiser5

Alan, Chris and Jeff support V3. Whenever they hit a bug that I think someone else should fix, or fail to get around to something fast enough (rare), I have vs do it. What more do you want?

What Chris and Jeff do that really irks me is add FEATURES not bugfixes to V3. As a result, almost all bugs for the last 2 years in V3 have been theirs, and mostly due to features being added. It also annoys me that they don't send every patch to a second person to test before sending it in and getting it accepted, like everyone else at Namesys does. What can I say, despite these gripes I am lucky to have them helping.

Features (with some exceptions) belong in major releases, not in bugfix releases.

Namesys follows an industry standard process of putting features in major releases and only bugfixes in the stable incremental releases. When we, or anyone else in the industry, do this, there are always users who complain that their feature that they need right now has been put into the major release and not released to them right now in the incremental releases. They usually use the phrase "you aren't supporting the (insert name of stable branch), you have abandoned us!"

Now Alan, as I understand it, it is usually YOU who is wishing that Linux would follow something closer to the release methodology that I just described. Maybe we have more commonality than you realize?

Hans

PS

Of course, with V4 we will have an interesting issue in that the plugins make it so easy to add features that there will be a temptation to let things dribble out one plugin at a time, with releases being more releases of plugins than releases of an entire filesystem. I haven't really thought through how to handle that best. I think I want a user to be able to define that he wants the mission critical server version of reiser4, and have that be an utterly stable body of code with the latest plugins excluded from it.

#194 lore raw

Re: reiser4 plugins

Quoted 4 lines · Hans Reiser

Alan, this is FUD. Our V3 fsck was written after everything else was,
for lack of staffing reasons (why write an fsck before you have an FS worth using). As a result, there was a long period where the fsck code was unstable. It is reliable now.

Was that more or less that about a year ago?

OG.

#195 lore raw

Re: reiser4 plugins

In reply to Hans Reiser
Quoted 8 lines · Hans Reiser

There is an area where we suffered from writing fsck last. When there are two leaf nodes with the same key range AND the bitmap cannot be trusted to tell us which is the valid one, we don't know which is the most recent, and pick arbitrarily. Also, if you store a backup of V3, and you don't compress it, and you wipe out the bitmap blocks and need to use fsck, we don't know what blocks are backup image and what blocks are the fs. We advise users to never store a V3 backup on V3 without compressing it.

Unfortunately, there are plenty of reasons why you might want to store a filesystem image on disk besides for backup purposes:

* Regression tests --- I have some 70+ small filesystem images used for e2fsck's regression test suite. (I am always amazed how many filesystem fsck programs don't have regression test suites.)
* Initial ram-disk images
* Image files for qemu or user-mode-linux

... and probably many more. None of these are safe to store on a reiserfs3 filesystem if you're worried about fsck being robust after a disk failure.

Funny thing. When I tell system administrators who have been around the block more than a few times about this particular "feature" of reiserfs3, they usually very quickly decide that it's time to switch to another filesystem.....

- Ted

#196 lore raw

Re: reiser4 plugins

fsck is better in V4 than it is in V3. Users should move from V3 to V4, as V3 is obsolete. I agree on that Ted.

I also agree that Ted did a great job with fsck.ext*.

V3 was where we learned. There are performance problems in V3 that I could only fix by writing V4. The balancing code for V3 was extremely difficult to modify because it understood all the internals of the item structures, and that is why we based V4 on plugins. V3 had a time where it was really useful, and the time when it was the only metadata journaling filesystem for Linux was its high point (thanks Chris), but its usefulness is leaving us very soon now with V4. In 12 months, after V4 has been pounded on by a few million users, few will want to make new installs onto V3 instead of V4.

It would be nice if we could concentrate on speeding that transition instead of flaming each other.

I would like to thank the ext* team, especially Ted, for a great filesystem that I used for many years while developing V3, that was crucial to the early success of Linux, and that is still useful to a great many.

Hans

#197 lore raw

Re: reiser4 plugins

In reply to Hans Reiser
Quoted 8 lines · Hans Reiser

Alan, this is FUD. Our V3 fsck was written after everything else was,
for lack of staffing reasons (why write an fsck before you have an FS worth using). As a result, there was a long period where the fsck code was unstable. It is reliable now.

People often think that our tree makes fsck less robust. Actually fsck can throw the entire internal tree away and rebuild from leaf nodes, and frankly that makes things pretty robust.

I did a series of tests well after resier3 had fsck that consisted of modelling the behaviour of systems under error state. I modelled random bit errors, bit errors at a fixed offset (class ram failure), sector 4 byte slip (known IDE fail case) and sectors going away.

Reiserfs didn't handle it anything like as gracefully as ext2. Its a pretty easy experiment to write the code for and the results are interesting.

#198 lore raw

Re: reiser4 plugins

Quoted 10 lines · Alan Cox
Quoted 8 lines · Hans Reiser

Alan, this is FUD. Our V3 fsck was written after everything else was,
for lack of staffing reasons (why write an fsck before you have an FS worth using). As a result, there was a long period where the fsck code was unstable. It is reliable now.

People often think that our tree makes fsck less robust. Actually fsck can throw the entire internal tree away and rebuild from leaf nodes, and frankly that makes things pretty robust.

Quoted 8 lines

I did a series of tests well after resier3 had fsck that consisted of modelling the behaviour of systems under error state. I modelled random bit errors, bit errors at a fixed offset (class ram failure), sector 4 byte slip (known IDE fail case) and sectors going away.

Reiserfs didn't handle it anything like as gracefully as ext2. Its a pretty easy experiment to write the code for and the results are interesting.

Maybe but I once checked some other error scenario. I generated (by mistake of course) dm table that lineary connected 3 times the same partition (instead of 3 different partitions). Both Reiser4 and Reiserfs3 gave a lot of errors while trying to use such device. Ext3 did not give single error and was hapily droping my data,

I agree that this is not very useful test case for disk problems but it shows that, at least, checks for trouble in Reiser4 are miles before those in Ext2/3. If only Reiser4 could print a note what I done wrong... ;-)

Grzegorz Kulewski

#199 lore raw

torturing filesystems [was Re: reiser4 plugins]

In reply to Alan Cox

Hi!

Quoted 5 lines
Quoted 5 lines

Alan, this is FUD. Our V3 fsck was written after everything else was,
for lack of staffing reasons (why write an fsck before you have an FS worth using). As a result, there was a long period where the fsck code was unstable. It is reliable now.

Quoted 8 lines
Quoted 3 lines

People often think that our tree makes fsck less robust. Actually fsck can throw the entire internal tree away and rebuild from leaf nodes, and frankly that makes things pretty robust.

I did a series of tests well after resier3 had fsck that consisted of modelling the behaviour of systems under error state. I modelled random bit errors, bit errors at a fixed offset (class ram failure), sector 4 byte slip (known IDE fail case) and sectors going away.

Do you have that code somewhere? Best I could do was overwriting random parts of partition with random data, you seem to have "more interesting" failures...
Pavel

Signature · 1 line
teflon -- maybe it is a trademark, but it should not be.
#200 lore raw

Re: reiser4 plugins

In reply to Alan Cox

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 5 lines · Alan Cox
Quoted 3 lines · David Masover
Quoted 2 lines

False argument. So does the pen, so do hinges on doors. Do you still have hinges on your doors - probably.

Quoted 8 lines
Quoted 2 lines

Indeed. Because there's nothing better -- not because I "like it the way it is".

I chose hinges carefully - there are better technologies than the generic hinge and there have been for many many years (plus cool stuff like magnetic doors). You of course aren't a door geek (nor am I I'd point out) but a computing one...

Right, but even if I was a door geek, changing hinges costs more than changing code. Now, if I were building a new house and I happened to know about other alternatives, I might not still be using hinges. Also, I leave enough doors open enough of the time that it's not a serious impact on my life.

Now, OS design is a place which does affect my productivity, so I do care about the design, and while it's hard to change that design, it doesn't actually involve buying stuff.

Quoted 7 lines
Quoted 3 lines

I think Hans (or someone) decided that when hardware stops working, it's not the job of the FS to compensate, it's the job of lower layers, or better, the job of the admin to replace the disk and restore from backups.

Most desktop users today don't have backups because there is no credible backup technology for 500Gb of data. They may have partial backups. Some

Bandwidth is getting faster. And I just found a nice site for backups called streamload.com. They don't seem to support rsync, and allow only 100 meg downloads, but unlimited uploads.

Few desktop users today really need to backup more than 50 megs of data.

Quoted 3 lines

things the fs can't deal with - if the disk goes boom then thats a lower level concern. Also certain bits like writing to spare blocks on a problem write are indeed handled drive level nowdays.

Right. And putting them in the FS is unneccesary bloat if you've got another mechanism for dealing with it. Anyone with 500 gigs of data can afford to do a little RAID, or at least some burned DVDs.

DVDs are cheap nowdays:
http://www.newegg.com/Product/Product.asp?Item=N82E16817502002

Ok, you might need two of those. Still, it's not much, unless that's 500 gigs of data which is actually changing pretty rapidly. Most users I know seem to burn CDs or DVDs of static things (like music) and only keep games and office-type files, plus copies of the stuff they've burned, on their hard drives. So, backup for most people I know = not throwing away game install disks + cron job to send < 50 megs of data to Streamload.

I agree it's nice to have a more corruption-proof filesystem. Convenient. But not absolutely necessary.

Quoted 8 lines
Quoted 4 lines

the issues are fixed, an entirely different crowd of benevolent dictators will come around and say that we can't get in because we change the VFS. At least some people on this list have said things to that effect.

There are four important issues I see here

Quoted 8 lines

1. It must work
2. It must be clean code that follows the kernel style
3. It must not break other stuff
4. It needs a maintainer who won't get bored 12 months later and only support reiser5

#3 is the VFS stuff and getting the VFS locking wrong or unclean is a *very* big deal because you'll cause corruption to non reiser4 users.

It's #2 and #3 that I'm worried about. #4 is a little unfair, and I can verify that #1 is satisfied.

And, by "worried about", I mean I'm worried about the attitude of other people. As far as I'm concerned, Reiser4 is mergeable now. But I'm not exactly an authority.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrzMMXgHNmZLgCUhAQLqMQ/+LeGfZlmMt8pog6StIbWj3ZtfSXXnOAtu Os6jMMITPL7QlGfD8ren6nExtS/zxAfua6eQmoc9Lk4qPSFGJsSj8s3APE6p+igr YLHSD6yKMW+sEwz+9d/jLI22R+Ct6x2AUGeYKbmJ4GnYYSeDMsJieeG/OJJscQFG ETjSDWYQM8Ba7YgiJBWfJFYfD9LuM2wE1yUY5zmqVlT1hKSEB6rKEpx+J4MpAj6H u19DCDoluVqTI/4XIFDjpHkwsNfYElCDe6dbdtgeHlDIjcgX8PRu2ZEVAhDjwG3H wlLc1V6lE/qznf4kUgWg3XAc6P2MbJAJd6S7xg5ifSaEzYE7sYwPrXAaPOh+BpNV P01T5eqsSoIbDQb1q1w686EVdQKXSXms2W21IFctHi60FBwfBIfLNwJ8I590MIhG SkhBj/LAnZrFDztf8Z16oBy0zNrVKLQBGkd3FlD8YDv8J7II266yv+aOVEvNp0Sw 5g8RnlxnUqnl1JCo7TRHLe3pknPXms9JISd/wHvjIPuxgOBSVw0nPDde+T/mdlOP 28ef3MDEDwc2cBe3xPLj1nXYjrLvw6zGrSR22IRYDrdZgqPtKjrUEj+0T6pmmHW/ NAH8Ck9zROvadnaB9/st4JJT74axRtwI1HFb4SMfo23vQ6za7RNIHKjDMVsAjtqV fmgsvO2QUo8=
=MlUY
-----END PGP SIGNATURE-----

#201 lore raw

Re: reiser4 plugins

Quoted 3 lines · David Masover

And, by "worried about", I mean I'm worried about the attitude of other people. As far as I'm concerned, Reiser4 is mergeable now. But I'm not exactly an authority.

Then why not listen to authorities, all of whom are saying "it's not ready yet"

Jeff

#202 lore raw

Re: reiser4 plugins

Quoted 2 lines · Jeff Garzik

Then why not listen to authorities, all of whom are saying "it's not ready yet"

What exactly is not ready Jeff? As I see it, this thread is 95% posturing, and almost no technical reasons for not merging. These "authorities"seem perfectly content with echoing the words of someone who skimmed the code and shot his mouth off without understanding it, and now these "authorities" just echo him because they like him and assume he must be right.

Actually, David's understanding of reiser4 is far deeper than those of anyone claiming to be an "authority". He understands what a plugin is. Wow. He does not have to be told that VFS does not currently support a notion of pluginids, so it is not duplicative. Wow. He does not have to be told that plugins are analogous to classes and what VFS currently has is analogous to instances. Wow. He gets it that reiser4 plugins disturb absolutely nothing in the rest of the kernel because we use the standard VFS interface. Wow. Does any of the rest of that echo chamber of authority understand those things? Seems to me that these "authorities" are just as capable of shooting their mouths off before doing their homework as anyone else is.....

David has used reiser4. Have you? Maybe you guys should try it. Maybe you should all have less faith in your ability to skim code and understand it instantly. Maybe you guys should actually read the technical parts of this thread rather than the flame parts only?

Maybe our community's social traditions are a bit lacking? Maybe saying that everyone else goes through this just means that changing the way we do things is needed even more?

Hans

#203 lore raw

Re: reiser4 plugins

Quoted 7 lines · Hans Reiser
Quoted 2 lines · Jeff Garzik

Then why not listen to authorities, all of whom are saying "it's not ready yet"

Quoted 6 lines

What exactly is not ready Jeff? As I see it, this thread is 95% posturing, and almost no technical reasons for not merging. These "authorities"seem perfectly content with echoing the words of someone who skimmed the code and shot his mouth off without understanding it, and now these "authorities" just echo him because they like him and assume he must be right.

I've already listed two rather large technical reasons.

When a vendor submits code, they know their [hardware | filesystem] best, but Linux kernel developers know the kernel code and kernel direction best.

There needs to be a meeting of the minds. Vendors that stick to "I know best" line of reasoning don't get very far in Linux.

Jeff

#204 lore raw

Re: reiser4 plugins

Quoted 2 lines · Jeff Garzik

I've already listed two rather large technical reasons.

They are?

#205 lore raw

Re: reiser4 plugins

In reply to Hans Reiser

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hans Reiser wrote:
[...]

Quoted 4 lines

David has used reiser4. Have you? Maybe you guys should try it. Maybe you should all have less faith in your ability to skim code and understand it instantly. Maybe you guys should actually read the technical parts of this thread rather than the flame parts only?

To be fair, I use it, but I haven't read much of it. I mostly read the whitepaper, and then read it again, several times, to make sure I groked it. Then I read the Future Vision paper, once, and completely failed to understand it. Now that I understand Spotlight, I might go back to it.

This is why I stay out of the deeply technical stuff, and frequently insist that I don't know what I'm doing and don't work here.

Quoted 3 lines

Maybe our community's social traditions are a bit lacking? Maybe saying that everyone else goes through this just means that changing the way we do things is needed even more?

The social traditions aren't uniform. In fact, if we're referring to all of open source, go hang out on irc.freenode.net#gentoo for a great community, whether or not you like the distro. If you want developers, there's not too many RTFM's and "I was coding bytecode engines when you were in dipers" attitudes in #perl, either.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrziPXgHNmZLgCUhAQKJiA//aIvOMGjwvorsU8YhIhR9jpqITFzR16/y YeXef0o7vX5UC6ws/eP3EQbl46/NbT8mth0l3x0OgXIkTha3edCPMtNUBuJ/ISad nYY17mNPCr5NWzyh5aL7Cw1DYBIVrwQ8dcVg7VbmnbutPyTxKnFxugABExk9MASQ +1n1FVmKRRl4cjte3qERnFN9crPxkLnGWWT5UneGo9tnxKCqd3VBcJQ23PYHsskF dyYmh9/YJaPMskFRK9oQzw9TAM740rwZvOOrQGO84fQkYnrgjrVxis3O6JYbVYzP tiQNC7kSE2HQoMwFratL+tTj2mpx2+Akmhb/XT738tDSNvpSIBhy2eGqXU2p61MF sZiz0kNiSpW2e2kUGYiDKtey9o7bTlQJJ1KmzLtVKqOhPgMpRkyR8PG7X9Sz0P// zLaETf02VHt+1stN9xKum9zYI3Ba0aOt3fhA+1RqBSszV9JdTYVA8/zekGSTYezx neEGJGMkZCRISY2hWQTmHYNVG55ie3cxGgKQdYRxM7wO6AMQID1074UMIqE2aPn5 qjGdtkLXtiLQ33gg7v5uAQeysyQxBZSEBI+uQSH3guXIjOX9XItMaqqygKNXB7ak zsZw8qguDVNHIAyPlHVBVHkCUEZ0hstdfYUU5gyMY+mLP0eq/HgSp5XEReX6RU0U 3KSielBMshs=
=ipn3
-----END PGP SIGNATURE-----

#206 lore raw

Re: reiser4 plugins

Quoted 5 lines · David Masover

The social traditions aren't uniform. In fact, if we're referring to all of open source, go hang out on irc.freenode.net#gentoo for a great community, whether or not you like the distro. If you want developers, there's not too many RTFM's and "I was coding bytecode engines when you were in dipers" attitudes in #perl, either.

I would develop for BSD instead if they were anywhere near Linux in marketshare. These attitudes, when the BSD guys had them, were why I chose Linux over BSD to develop for. Now Linux has the market share,
and they act like the old BSD guys used to. Sad. Somebody else will
come along and challenge Linux someday. I wonder if Apple is a better
social environment for developers these days than Linux? It would be fun to work with Steve Jobs, he has such a sense of vision and a delight in new things. He hires good people too; Dominic Giampaolo is really sharp.

Hans

#207 lore raw

Re: reiser4 plugins

Quoted 5 lines · Hans Reiser

I wonder if Apple is a better
social environment for developers these days than Linux? It would be fun to work with Steve Jobs, he has such a sense of vision and a delight in new things. He hires good people too; Dominic Giampaolo is really sharp.

And trade freedom for this? In the current situation you can release code even if the powers that be do not agree. Others can choose to integrate and distribute your code, .. they have and they will..

If you were at apple and some authority decided against a feature you and your team would likely not be permitted to develop it, and if developed it would not be distributable.

The situation right now is somewhat unfortunate, but it is not unsolvable. Trading away freedom is seldom a good deal.

#208 lore raw

Re: reiser4 plugins

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 10 lines · Gregory Maxwell
Quoted 5 lines · Hans Reiser

I wonder if Apple is a better
social environment for developers these days than Linux? It would be fun to work with Steve Jobs, he has such a sense of vision and a delight in new things. He hires good people too; Dominic Giampaolo is really sharp.

And trade freedom for this? In the current situation you can release

[...]

Quoted 2 lines

The situation right now is somewhat unfortunate, but it is not unsolvable. Trading away freedom is seldom a good deal.

The irony is killing me.

Namesys is, I'm told, still quite in debt. Apple has money. From a certain point of view, it is a good deal...

...but to have your cake and eat it, there's Google... -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrz+DngHNmZLgCUhAQKd+A//Vif3OzRDdPHuwGVfypfJQ9ptaPUxcv/W qR8GJfGL7FNP3cGD4PJMcCJPU0tnjDVWtSDmBRP+qhWnxPq36LcLF7fIznaGcv1K zkMS7yA8UQrDtKfFwHRbi3EvOhEKsrnkCKPnuZ1SRa3aNQD2sJ89R+0641PbHnpY 2g3G7qyc2lIep59I6Eb+b4X4ZHal5BnWf3nUTFrfCt8N9pmQcWxdeTdhw0i2g+A2 UmB0nG2wL1SOjnsT8M/pFwFdF5Wft+wXrEKD5j6FCzJ/J3TRBJ90iAgWaDb5IK8b 80EcqNSKYKDzqHly0tM/m2NMDP1vwhsOPCPZlLOKoB1YvBbSuOLtpzWCOcpDuFox PuIgdBjyHVuEeY+zdPn2e2UiP5FsYcocT7VJMAwF/JJ+GVdRDci8UDqLtx1iFQ4R ukBgJ+9XbCMGOjzRe/GoCcxmsoHGqkcsbK0vaAhCxejQkwwEwr04ycj7BPXKAVMW nmmKqCw6Y+8FP7oAoZC3wsR9frF+qQbEQvlmbDQ9oKm0EzRM871V0si3imG3gyyF c37Jg1HvgxfYCrvqjkxKzHU3csUvZH9oZDfax1ThvH8D2a3pJuo1oJLdbGseedlh k8S7YxCm6hXA82RSHGV8ljo453S4j5QO42sSjlMihEfb6Ft6OsXF1P7Hqjw93uF7 s69+sVS6Rho=
=2aOi
-----END PGP SIGNATURE-----

#209 lore raw

Re: reiser4 plugins

In reply to Hans Reiser
Quoted 9 lines · Hans Reiser

I would develop for BSD instead if they were anywhere near Linux in marketshare. These attitudes, when the BSD guys had them, were why I chose Linux over BSD to develop for. Now Linux has the market share,
and they act like the old BSD guys used to. Sad. Somebody else will
come along and challenge Linux someday. I wonder if Apple is a better
social environment for developers these days than Linux? It would be fun to work with Steve Jobs, he has such a sense of vision and a delight in new things. He hires good people too; Dominic Giampaolo is really sharp.

I hate to jump in here but this is just too much!

Hans, I remember well the bellyaching you did when you insisted on the Linux developers merging Reiserfs V1 into the kernel. Did you not learn from *that* experience? You have taken *some* criticism here which may *not* have been flames at you until you flamed *first*. You have been told by at least 2 developers here what you must do *first* before consideration of merging V4. As of this writing YOU HAVE NOT ANSWERED! You obviously have not even read all articles posted here. When Jeff Garzik said "I've already listed two rather large technical reasons." You answered "They are?". You just continue your insane bickering, now followed by picking up your marbles and leaving. Grow up.

BTW, I could care less whether V4 gets merged or not as I am *not* a developer.

Bob

Signature · 1 line
Bob R. Taylor <[email protected]>
#210 lore raw

Re: reiser4 plugins

In reply to Hans Reiser
Quoted 6 lines · Hans Reiser

What exactly is not ready Jeff? As I see it, this thread is 95% posturing, and almost no technical reasons for not merging. These "authorities"seem perfectly content with echoing the words of someone who skimmed the code and shot his mouth off without understanding it, and now these "authorities" just echo him because they like him and assume he must be right.

I'm not one of those "authorities" but I doubt reiser4 will be merged at all with that attitude.

#211 lore raw

Re: reiser4 plugins

Quoted 12 lines · --- Diego Calleja
Quoted 6 lines · Hans Reiser

What exactly is not ready Jeff? As I see it, this thread is 95% posturing, and almost no technical reasons for not merging. These "authorities"seem perfectly content with echoing the words of someone who skimmed the code and shot his mouth off without understanding it, and now these "authorities" just echo him because they like him and assume he must be right.

Quoted 2 lines

I'm not one of those "authorities" but I doubt reiser4 will be merged at all with that attitude.

Spot on. Somebody hasn't read his Linus. It seems to me that anytime I hear whining about how unfair the system is, how arbitrary patch rejection is, the flaw is either a lack of social competence, or failure to follow the submission guidelines. Lately the number three reason, "Linus forgot," happens less and less. And linux developers are offering to help Reiser development fix what's wrong! That's above and beyond.

If Hans wants to take his ball and go home, let it be understood that it is not because of a failure in the linux process.

People that don't want flames, drop out of flame wars and do constructive work. People that want to talk ideology, start flame wars and perpetuate them. If you're having a code argument and you can go four posts without referring to actual code, you're no longer having a code argument. Non-demonstrable merit is not merit. Benchmarks can be necessary -- but are never alone sufficient -- merit to merge to mainline.

Hans: consider that enough linux people like your stuff enough to have this argument. Performance has long since ceased to be at issue. Linux is a thing in itself. If Reiser4 is a thing in itself, and not merely a means to a further release point, you will care enough about it to merge it usably and maintainably into the kernel. Should you obsolesce 4 as you have done to 3, someone else can care enough about it, and do something with it. Features are added to Reiser3 because it is a thing in itself now that it is in the kernel. Its users care. Its maintainers, even if they still work for Namesys, care. Listen next time you hear someone talk about "vendor-abandoned projects" continuing to function through community development.

Alan has said, "Free software is always late." Free software follows the itch. If Hans wants linux to have Reiser4, Reiser development will work with linux development to become compliant and explain/adapt any additional functionality. Linux does not abjectly need Reiser4 in-tree. Reiser4 is ultimately Hans's scratch for Hans's itch. It will be the more effective and widely usable once it is stable in-tree, but the mainline kernel developers aren't the ones with the itch. Protocols exist in social systems as a means to constructively enlist the help of people outside the problem. The protocols here are well established. This is a patch meritocracy. Failure to comply further increases the lateness of Reiser4 in mainline kernels. Ego involvement is noise.

My observations being what they are, may I suggest that this thread die, in favor of more eyes and in-depth current code analysis help in "Re:
reiser4 merging action list"? Higher utility function; higher S:N. Flames can always go down with the ship.

Frosty
Not a programmer, merely literate.

No, I don't usually respond to trolls, but this thread, from this man, itches me. It's not worthy of his brilliance, which may be his tragic flaw.

#212 lore raw

Re: reiser4 plugins

In reply to David Masover

["Followup-To:" header set to gmane.linux.kernel.] I gmane.linux.kernel, skrev David Masover:

Quoted 8 lines
Quoted 2 lines

Most desktop users today don't have backups because there is no credible backup technology for 500Gb of data. They may have partial backups. Some

Bandwidth is getting faster. And I just found a nice site for backups called streamload.com. They don't seem to support rsync, and allow only 100 meg downloads, but unlimited uploads.

Few desktop users today really need to backup more than 50 megs of data.

That gives tedious manual work.. and btw, won't save you if you from loosing stuff from when the backup was made until now.

Quoted 10 lines
Quoted 3 lines

things the fs can't deal with - if the disk goes boom then thats a lower level concern. Also certain bits like writing to spare blocks on a problem write are indeed handled drive level nowdays.

Right. And putting them in the FS is unneccesary bloat if you've got another mechanism for dealing with it. Anyone with 500 gigs of data can afford to do a little RAID, or at least some burned DVDs.

DVDs are cheap nowdays:
http://www.newegg.com/Product/Product.asp?Item=N82E16817502002

Again lots of manual work.. I actually have a DAT-station.. but I'm not getting it used. People have DVD-burners, but many don't get time to do a backup anyway. A Copy-On-Write feature in the filesystem would save the average dataloss situation todag (for home users). Where they accidentally deletes stuff.

Quoted 1 line

Streamload.

Why, when it could be quick and transparent. And Linux is used many places where you cant let data out-of-the-house of where bandwidth "sucks". The waste-space in my diskdrives increases everyday .. and i fill up with a tar-ball of the system every now-and-then, but it would definately be better suited and more effecient (save me more times) done directly in the filesystem.

Quoted 2 lines

I agree it's nice to have a more corruption-proof filesystem. Convenient. But not absolutely necessary.

Thats called raid, we have that allready. But raid won't help for and:
rm /etc/passwd, a Copy-On-Write filesystem (not-snapshot) would. Used on a mirrored raiddisk, with enough space on the disk, it would actually guard you from allmost anything but getting the computer stolen.

Totally unrelated to reiser4 but a feature that would be nice to have in "any" filesystem.

Jesper

Signature · 1 line
./Jesper Krogh, [email protected]
#213 lore raw

Re: reiser4 plugins

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 10 lines · Jesper Krogh

["Followup-To:" header set to gmane.linux.kernel.] I gmane.linux.kernel, skrev David Masover:

Quoted 7 lines
Quoted 2 lines

Most desktop users today don't have backups because there is no credible backup technology for 500Gb of data. They may have partial backups. Some

Bandwidth is getting faster. And I just found a nice site for backups called streamload.com. They don't seem to support rsync, and allow only 100 meg downloads, but unlimited uploads.

Quoted 5 lines
Quoted 1 line

Few desktop users today really need to backup more than 50 megs of data.

That gives tedious manual work.. and btw, won't save you if you from loosing stuff from when the backup was made until now.

Manual?

Try scripting. For me, that's a tar command involving /home, /etc, and about one or two other files, with a few excludes, like /home/shared/video.

Quoted 12 lines
Quoted 10 lines
Quoted 3 lines

things the fs can't deal with - if the disk goes boom then thats a lower level concern. Also certain bits like writing to spare blocks on a problem write are indeed handled drive level nowdays.

Right. And putting them in the FS is unneccesary bloat if you've got another mechanism for dealing with it. Anyone with 500 gigs of data can afford to do a little RAID, or at least some burned DVDs.

DVDs are cheap nowdays:
http://www.newegg.com/Product/Product.asp?Item=N82E16817502002

Quoted 5 lines

Again lots of manual work.. I actually have a DAT-station.. but I'm not getting it used. People have DVD-burners, but many don't get time to do a backup anyway. A Copy-On-Write feature in the filesystem would save the average dataloss situation todag (for home users). Where they accidentally deletes stuff.

A lot of the people I know keep stuff on their DVDs, like movies and music, so they can carry them around. And the rest of it is the 50 megs.

Quoted 9 lines
Quoted 1 line

Streamload.

Why, when it could be quick and transparent. And Linux is used many places where you cant let data out-of-the-house of where bandwidth "sucks". The waste-space in my diskdrives increases everyday .. and i fill up with a tar-ball of the system every now-and-then, but it would definately be better suited and more effecient (save me more times) done directly in the filesystem.

Streamload is quick and transparent for me. I put files on the fileserver, it tars them up and uploads them via Streamload's perl client.

Quoted 11 lines
Quoted 2 lines

I agree it's nice to have a more corruption-proof filesystem. Convenient. But not absolutely necessary.

Thats called raid, we have that allready. But raid won't help for and:
rm /etc/passwd, a Copy-On-Write filesystem (not-snapshot) would. Used on a mirrored raiddisk, with enough space on the disk, it would actually guard you from allmost anything but getting the computer stolen.

Totally unrelated to reiser4 but a feature that would be nice to have in "any" filesystem.

Not totally unrelated. COW has been discussed. I don't remember if the low-level stuff was done, but the main complaint was a lack of a copy system call.

And RAID is an argument for Reiser4's attitude that it's not the job of the FS to be corruption-proof.

Still, it's far easier to avoid deleting stuff than to avoid disk failure. First priority is to get the stuff OFF the machine.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr0LxXgHNmZLgCUhAQKKBBAAiAicaEKY42s0WctOcLg/m/ciVtphQ1jY rChlThrsRCl4xAE45GgAu4P1ZdEYGwI1574W9z2j8EpbdnghX8tBHFUDSG1/K3+f 8Ud0zyMaI46k+D/IzkXS1ENYDj1PGmPAuVbM2pAa3JW0UOMzvKRsSADewxAW7OdY V9fazSYu6l7Sn64XJxpZmXs9fkElXkqaNu/2N5d6rdH8hMmLhxs8HcsbAaJ7ch87 lz2RMruQ4Keh6H6HTHHvmoYog6XakwkD0pOY0efFZolO/EZFnmIlS9VHRIyb6mls 2xKPABDh9Qq0qQxpATgmXnVI9oh9qgrp4wqQ8nTQfEnhL5fvfBTXzJgfjOJiqXUy vX4K/PP+9wzQNoqhTj4g11JBT8ilemsA4U+gbr82qSvvkOXrHkgGTQe6wgUyo6GZ R8Ui7/jmAvNw+RvfL/p0s0s3e/EimUC7ASvN64z6z77wqNH0uUfUwRmD9SL8TbTf jPdFvcd65F84T4gXphT4Dhwjs4fVjZUq3BqB+zMl3lKJP6pJfc5bqpjvqc2Rg1Yv jfpvk3gihG7YN68IgZV+9IHJG6d5P05gi/YMqu75JDkqTXe2VznOXXQp4d58PhO0 ZKJnvAfr/xKrnuwgDUjgfaHThgquyMiC95hMo4IlSEfUquKPFGY4c9k0VKQHduqc lcZOnfkvar0=
=olqO
-----END PGP SIGNATURE-----

#214 lore raw

Re: reiser4 plugins

In reply to David Masover
Quoted 2 lines · David Masover

Right, but even if I was a door geek, changing hinges costs more than changing code. Now, if I were building a new house and I happened to

Probably not, programmers are expensive 8)

Quoted 4 lines

DVDs are cheap nowdays:
http://www.newegg.com/Product/Product.asp?Item=N82E16817502002

Ok, you might need two of those. Still, it's not much, unless that's

At least four because the media decay patterns are not well understood. So you need raid DVD too 8)

Quoted 8 lines
Quoted 5 lines

1. It must work
2. It must be clean code that follows the kernel style
3. It must not break other stuff
4. It needs a maintainer who won't get bored 12 months later and only support reiser5

It's #2 and #3 that I'm worried about. #4 is a little unfair, and I can verify that #1 is satisfied.

Please review the 2.5 history of reiserfs3 if you believe so

#215 lore raw

Re: reiser4 plugins

In reply to David Masover
Quoted 3 lines · David Masover

I was able to recover from bad blocks, though of course no Reiser that I know of has had bad block relocation built in... But I got all my files off of it, fortunately.

My experience shows that you've been very, very lucky. I hope r4 is better in that regard.

If you want to try with r3, take a well-used partition[1] and copy it at block level to another partition or a file. Then zero some random spans of blocks in the copy and reiserfsck --rebuild-tree it. My experience is that you'll usually get the files names and directory tree but their contents will have been scattered all over the place.

OG.

[1] I suspect a minimum of fragmentation is in order to see the problem

#216 lore raw

Re: reiser4 plugins

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 9 lines · Olivier Galibert
Quoted 3 lines · David Masover

I was able to recover from bad blocks, though of course no Reiser that I know of has had bad block relocation built in... But I got all my files off of it, fortunately.

My experience shows that you've been very, very lucky. I hope r4 is better in that regard.

I was on Reiser4. And as Hans is loudly protesting, v3 seems to be stable now, including being somewhat resilient to disk breakage.

But I do better backups now.

Quoted 5 lines

If you want to try with r3, take a well-used partition[1] and copy it at block level to another partition or a file. Then zero some random spans of blocks in the copy and reiserfsck --rebuild-tree it. My experience is that you'll usually get the files names and directory tree but their contents will have been scattered all over the place.

Sometime I'll try that. I do remember mounting read-only *before* I tried the --rebuild-tree, because I didn't want to give it a chance to screw up more that it already was.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrzG3XgHNmZLgCUhAQKayQ//Z4R/1lKkjaiHfIrIpVXmLrGfcGUuVXLG FeQb+50orlA6pfaHDJ5JkJKKyHnTsnNSvn+VHdxqvz25/ijWDoewe368syXSggOA JkSUSvdQlcKn+dnpT8tPFY8OU3Ve87zb53fAI9J145DRLpXXmWjW5/R1OUzlPnmr +nn2n32UvbSyUj8VAM9ECtJ3+HaxXKm0ASbQLKgLoLmJ5bO/u2xml7rGtB1kdaCr xJIxugAaWnA/sytoK2vIsdnH4QFQqbuqW4bdHa3ziORS1w/DQ08PMFtxu/3ChKVQ 5d0cGtMsZkVIz+k4zE+3RD2QXZVLRvZCTwPqQciitvXCRqz3d9Wc73D+fC8FwLKx t7aARQKprPDA/bgg4SPAoJSVxZMTJDATuJ5f6XrpDB/FVUmHYQfqNLhOD/Vl5IAA 9aZCPjDXMknClydasw/SiRu3DUiOIKYlXXUN1Tnsx4l9lbVJpca25TBem+0Val2P IsruPiJ8L3XztRNwQJOY4RGH1CMobPnwEA4F9PFKqVF+wwJS5sNNqbkoD7XMrOT6 UYn3KC0DixgVCkl75nP6ORs8F9LjIuDtFd/+KNQC+AlcU5MX+BeaJD1QhT5G/1qd GjRcQaN+95jVxLVjkqnb552dQ7+CMt/DzszFDtfKlJepobflVMtX1hoZ0phudAgR PqjDS3tIROk=
=lkfJ
-----END PGP SIGNATURE-----

#217 lore raw

Re: reiser4 plugins

In reply to Olivier Galibert

Hi!

Quoted 4 lines
Quoted 3 lines

I was able to recover from bad blocks, though of course no Reiser that I know of has had bad block relocation built in... But I got all my files off of it, fortunately.

Quoted 8 lines

My experience shows that you've been very, very lucky. I hope r4 is better in that regard.

If you want to try with r3, take a well-used partition[1] and copy it at block level to another partition or a file. Then zero some random spans of blocks in the copy and reiserfsck --rebuild-tree it. My experience is that you'll usually get the files names and directory tree but their contents will have been scattered all over the place.

Actually it would be nice to have a script to automate this, and test various filesystems for robustness :-). I done something similar, and it uncovered few bugs in ext2fsck and vfatfsck; perhaps it is time to try it on other filesystems, too?
Pavel

#!/bin/bash
#

# fscktest
#
# Usage: 
#	 Make sure output is logged somewhere
#        First, run fscktest -p as root
#	 Then you can run fscktest as normal user...
#

prepare() {
SIZE=100000
echo "Creating file..."
cat /dev/zero | head -c $[$SIZE*1024] > test echo "Making filesystem..." /sbin/mkfs.$FS test
echo "Mounting..."
mount test -o loop /mnt || exit "Cant mount" echo "Copying files..."
cp -a /bin /mnt
cp -a /usr/bin /mnt
cp -a /usr/src/linux /mnt
echo "Syncing..."
sync
echo "Unmounting..."
umount /mnt
echo "Moving..."
mv test fsck.okay
echo "All done."
}

FS=ext2
if [ .$1 == .-p ]; then
prepare
exit
fi
RUN=0
while true; do
RUN=$[$RUN+1]
echo "Run #$RUN"
echo Preparing...
cat fsck.okay > fsck.damaged
echo Damaging...
dd if=/dev/urandom of=fsck.damaged count=10240 seek=7 conv=notrunc cp fsck.damaged fsck.test
echo First check...
/sbin/fsck.$FS -fy fsck.damaged
RESULT=$?
if [ $RESULT != 1 -a $RESULT != 2 -a $RESULT != 0 ]; then
echo "Fsck failed in bad way (result = $RESULT)" exit
fi
echo Second check...
/sbin/fsck.$FS -fy fsck.damaged
RESULT=$?
if [ $RESULT != 0 ]; then
echo "Fsck lied about its success (result = $RESULT)" exit
fi
done

Signature · 1 line
teflon -- maybe it is a trademark, but it should not be.
#218 lore raw

Re: reiser4 plugins

In reply to David Masover
Quoted 4 lines · David Masover

I was able to recover from bad blocks, though of course no Reiser that I know of has had bad block relocation built in...

there was a patch somewhere. Vitaly, please comment.

#219 lore raw

Re: reiser4 plugins

Quoted 9 lines · Hans Reiser
Quoted 4 lines · David Masover

I was able to recover from bad blocks, though of course no Reiser that I know of has had bad block relocation built in...

there was a patch somewhere. Vitaly, please comment.

http://www.namesys.com/bad-block-handling.html describes how reiserfs handles bad blocks.

Signature · 2 lines
Thanks,
Vitaly Fertman
#220 lore raw

Re: reiser4 plugins

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 7 lines · Vitaly Fertman
Quoted 5 lines · Hans Reiser
Quoted 1 line · David Masover
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

I was able to recover from bad blocks, though of course no Reiser that I know of has had bad block relocation built in...

there was a patch somewhere. Vitaly, please comment.

http://www.namesys.com/bad-block-handling.html describes how reiserfs handles bad blocks.

Anything like this for v4?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQsBqhHgHNmZLgCUhAQLnqQ/8CNGlg/b2rS5TJi6Qzbi5E3eQMDXpU6GO bQZCrZ60dhPGI0z5AqM+kpd4tfMqvF9pW83Pa4xPN9Snvs4RxJt79JutRss5JByF ZrBdwt2Whq0Hmv5MxbIlCKof92thg3AL7HVmuKVgApAy85cOLH3C0CTMS00KjHqn oVDO6gAYlZxJXd/Z+VqCILRTorcTJQXi7NaZy0ptSYPZlna/1JL+0JUOKOAQ8U++ 8xK/ETwMwnWRd0/yZfrjE7Y1hU1bFlM5QAHKoRHnXfXEqXNm5eg+idsfCbrv80uw +QuQy46Xl0puumcB1G6kp+WLboQe+u4fCn85osiWo8OzVcU4Q4hkqCjV6kOo6WH5 VYn/W3kEgdEpgxnkuDEG4uxiootzpUPUVLpVsK2GSqo5pdiNz63IGvt40WYQOnkc I6/YPDTlHdmrFbxpASY05gMdtHYDEkvI7cw8MBeaK35Fr7S5VvCeRkEFI48CZHTw Yz+wljq6v22tPmkdKZByXqIY+8A5hgSf2eBXGG4HBL9ogBFbj+IkueqhRQ9Z023J EL7qYVlO0EDmpxb+8MbDYvugMXk6PlzebZLNu4Zw7S0sbkZFUhdkCQS3bnn22kJ4 UxYfsxy7fTUl5Rbdk+TzPbUX39U/fO9seZi6euEt6Dc3ak4g9lVnnzIo8GkwPBNF 7KRZCgN3wCw=
=7bqs
-----END PGP SIGNATURE-----

#221 lore raw

Re: reiser4 plugins

Quoted 11 lines · David Masover
Quoted 10 lines · Vitaly Fertman
Quoted 8 lines · Hans Reiser
Quoted 3 lines · David Masover

I was able to recover from bad blocks, though of course no Reiser that I know of has had bad block relocation built in...

Quoted 7 lines
Quoted 5 lines
Quoted 1 line

there was a patch somewhere. Vitaly, please comment.

http://www.namesys.com/bad-block-handling.html describes how reiserfs handles bad blocks.

Anything like this for v4?

in todo for v4, not implemented yet.

Signature · 2 lines
Thanks,
Vitaly Fertman
#222 lore raw

Re: reiser4 plugins

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 18 lines · Vitaly Fertman
Quoted 16 lines · David Masover
Quoted 14 lines · Vitaly Fertman
Quoted 9 lines · Hans Reiser
Quoted 2 lines · David Masover

I was able to recover from bad blocks, though of course no Reiser that I know of has had bad block relocation built in...

there was a patch somewhere. Vitaly, please comment.

Quoted 7 lines
Quoted 4 lines
Quoted 2 lines

http://www.namesys.com/bad-block-handling.html describes how reiserfs handles bad blocks.

Anything like this for v4?

in todo for v4, not implemented yet.

Is it significantly different that there'd be another doc I can read?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQsGhjXgHNmZLgCUhAQJaeg/7BeQVRKc0m0AEKiwp6B/N/R6/32JcIvk9 0jZq3A6wYsSxmRSDfId+FVoHApAoZldOtw5CZ7W3TwUJBxzGwFZ7IYM4mQaZ5eRa 7CdAoX9LrG85sY/2M9qypE5cW6BQ/0DJFPDOiree/+6mNomg8uOUJ6FeO93j2JW3 jdPrWqsmYBPSDo4vKha+xoAZpCH/sMLwDDBUIFfC2iz4B4yTxYNfEX+mJ4T2mEls 3YqSiTMkgnFbsVTdUiIjqVJY+jJ9ALhDKdc/hkNNrNir3Iib6aNbg9J9J5cDgNHU zzvNyp6WsCgQjtCvfMojVewKAMu94eg1sXt6ESNz4dKbdPBSdo+lBnATntnMjFCN 7nS9K/kKcTKwSVQjQdyEhbm51C9SXYLxj4CMcru9ZyVtdlyzCLhBda1FvaxWbatJ iGCh8tg43Xkd1N4H+LlTQpJrfbkJBXu4U9oYIuNSQuolMfz6IbJqcfNzV0hm9drp lGLODFSFEWFqHUgPqLymk8/pBW9xP4XYwwWFcY8x0Dw4UxpWRB43EZ8Nydg4PBEk SrLnq1FNSP+QLHZvWwF+8vpWj1jyeiBU2hFpt/p9crU5uy4OWGbKEh8GuEHFgp9P BdKzp4iwTFwgjKR/uM3JnCfN4ct0yyyBZVMNf0N8w2Idu4OkBiS2edg/70A8/OqV U0JVANH5qxc=
=sd/Q
-----END PGP SIGNATURE-----

#223 lore raw

Re: reiser4 plugins

In reply to David Masover
Quoted 6 lines · David Masover

But, there are some things Reiser does better and faster than ext3, even if you don't count file-as-directory and other toys. There's nothing ext3 does better than Reiser, unless you count the compatibility with random bootloaders and low-level tools.

In fairness, there are some things that I am aware of. In Reiser4 fsync performance needs optimizing, truly random modifications and anything else that write twice logging is best for needs optimizing, workloads dominated by dirtying more mmap'd pages than can fit into RAM need optimizing.

Quoted 12 lines
Quoted 7 lines
Quoted 4 lines

Not everyone will want
to reformat at once, but as the reiser4 code matures and proves itself (even more than it already has),

I for one have seen mainly people with wild claims that it will make

their

Quoted 2 lines

machines much faster, and coming back later asking how they can recover their thrashed partitions...

Quoted 7 lines

You know how many I've had thrashed on Reiser4? Two. The first one was with a VERY early alpha/beta, and the second one was when I dropped a laptop and the disk failed.

And it does make certain things faster. For one thing, "emerge sync" on Gentoo is twice to four times as fast, and /usr/portage is 75% as big, as on ReiserFS (3).

Wow. You know, Nikita complains about my benchmarks, but what I usually hear from users is that their performance when they use their favorite app is mostly similar to my benchmarks.

I would be really happy if our compression code was working before we went in. Half the space and thus twice the speed to disk. Edward could use some help with it. If only we could work on that plus fsync plus mmap dirtying plus random modification optimized journaling rather than these flamefests.

Hans

#224 lore raw

Re: reiser4 plugins

In reply to David Masover
Quoted 11 lines · David Masover
Quoted 9 lines · Horst von Brand
Quoted 2 lines · David Masover
Quoted 1 line · Hans Reiser

Jeff Garzik wrote:

[...]

Quoted 3 lines
Quoted 3 lines

You missed his point. He is saying ext3 code should migrate towards becoming reiser4 plugins, and reiser4 should be merged now so that the migration can get started.

Quoted 7 lines
Quoted 5 lines
Quoted 3 lines

Sort of.

I think ext3 would be nice as a reiser4 plugin.

What for? It works just fine as it stands, AFAICS.

So does DOS.

I'm sorry?

Quoted 1 line

Do you use DOS?

Good riddance, no! Not for something like 15 years.

Quoted 1 line

I don't even use DOS to run DOS programs.

Haven't seen one in quite some time, to be honest. Oops, sorry, I just lied. Booted it from floppy to run a program to whack the BIOS password on a PC some two weeks ago. Only use in a long time.

Quoted 1 line

"Ain't broke" is the battle cry of stagnation.

I see it as the battle cry of those that are looking for /real/ problems to solve.

Quoted 1 line

But, there are some things Reiser does better and faster than ext3,

Yes, I've heard it is supposed to be faster on huge directories, and doesn't run out of inodes. And it is more efficient spacewise on small files. But then again, space is extremely cheap today...

And again, on a list around here I've seen several cries for help with completely hosed filesystems, all ReiserFS. No solution has ever come forth.

Quoted 4 lines

even
if you don't count file-as-directory and other toys. There's nothing ext3 does better than Reiser, unless you count the compatibility with random bootloaders and low-level tools.

For me, those are quite critical...

Quoted 11 lines
Quoted 7 lines
Quoted 3 lines

Not everyone will want
to reformat at once, but as the reiser4 code matures and proves itself (even more than it already has),

I for one have seen mainly people with wild claims that it will make their machines much faster, and coming back later asking how they can recover their thrashed partitions...

You know how many I've had thrashed on Reiser4? Two. The first one was with a VERY early alpha/beta, and the second one was when I dropped a laptop and the disk failed.

OK. Know how many I thrashed with ext2/3? I remember 3, could have been as many as 5. One was due to a failed disk, another one because of DMA to a disk causing slow corruption. Another one I believe was due to a odd kernel compiled with a snapshot gcc a long time back, plus power loss at the wrong time. And that is from using ext2/3 since they were in early beta. On several machines at the same time, over years. I'd have to say that there isn't much of a difference.

Quoted 3 lines

And it does make certain things faster. For one thing, "emerge sync" on Gentoo is twice to four times as fast, and /usr/portage is 75% as big, as on ReiserFS (3).

That can't all be due to on-disk format.

Quoted 8 lines
Quoted 4 lines
Quoted 2 lines

the ext3 people may find themselves
wanting some of the more generic optimizations.

They'll filch them in due time, don't worry.

Duplication of effort. With plugins, we can optimize the upper layers of ALL filesystems, regardless of the lower layers, in such a way that it is optional.

Generic optimizations how, if they need VFS support?!

Quoted 2 lines

I'm sure it's far easier to write a Reiser storage
plugin than a brand new FS.

Comparing apples and oranges tells you what?

Quoted 4 lines

Eventually, once competition is only based on storage format, we could end up with just one format. Just one filesystem! (except for fat/ntfs/iso/udf/network...) And in the open source world, sometimes a single product is a good thing.

Sorry, I don't think this will come to pass in our lifetime, if ever. There are different requirements, and the way to cater to them is different solutions. That has always been what Linux (as opposed to the propietary and even some open source systems) is all about...

Quoted 10 lines
Quoted 9 lines
Quoted 6 lines

But, I don't think that will realistically happen at all.

Instead, what will probably happen is that once Reiser4 is in the mainstream kernel, it will become more popular and noticable. Other FSes will take note. ext3 people may decide they want file-as-directory,

That idea is even much older than Linux itself, and no other (Unix) filesystem has implemented it. Ever. Wonder why...

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

and vfat people may decide they want cryptocompress,

I'm sure they don't, because it is mostly for Windows and assorted devices (pendrives, digital cameras, ...) compatibility.

I, for one, would like to use a pendrive and have certain files be encrypted transparently, only for use on Linux, but others be ready to transfer to a Windows box.

And that would surely break Windows compatibility (because you have to keep the data on what to encrypt one the filesystem itself). Besides, having pgp, or gpg, or crypt, or my own whacky encryption proggie do the job in /userland/, and not shoving into the kernel and so down the next user's throat, is in the end a largeish part of what Unix is all about for me.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#225 lore raw

Re: reiser4 plugins

Quoted 7 lines · Horst von Brand
Quoted 3 lines · David Masover

I, for one, would like to use a pendrive and have certain files be encrypted transparently, only for use on Linux, but others be ready to transfer to a Windows box.

And that would surely break Windows compatibility (because you have to keep the data on what to encrypt one the filesystem itself).

Easily solved, using the same method OS/2 added Extended Attribute support to FAT. Of course, things may break if you try to manipulate the encrypted file under Windows (e.g. deleting those files would leave some garbage blocks behind), but breakage should be minor and relatively easy to fix.

Quoted 4 lines

Besides, having pgp, or gpg, or crypt, or my own whacky encryption proggie do the job in /userland/, and not shoving into the kernel and so down the next user's throat, is in the end a largeish part of what Unix is all about for me.

Meh. It's nice to have encrypting to be transparent. Unless your editor supports gpg/crypt/etc., you'll need to decrypt the file to a temporary area, edit the file, and then re-encrypt. Much easier to make a mistake and lose data, or leave traces of the cleartext data.

Besides, the encryption would be optional, and hence not being shoved down the user's throat. If you don't like it, don't use it.

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#226 lore raw

Re: reiser4 plugins

In reply to Horst von Brand

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 37 lines · Horst von Brand
Quoted 19 lines · David Masover
Quoted 17 lines · Horst von Brand
Quoted 3 lines · David Masover
Quoted 1 line · Hans Reiser

Jeff Garzik wrote:

[...]

Quoted 7 lines
Quoted 3 lines

You missed his point. He is saying ext3 code should migrate towards becoming reiser4 plugins, and reiser4 should be merged now so that the migration can get started.

Sort of.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

I think ext3 would be nice as a reiser4 plugin.

What for? It works just fine as it stands, AFAICS.

So does DOS.

I'm sorry?

Quoted 1 line

Do you use DOS?

Good riddance, no! Not for something like 15 years.

[...]

Quoted 5 lines
Quoted 1 line

"Ain't broke" is the battle cry of stagnation.

I see it as the battle cry of those that are looking for /real/ problems to solve.

I'll refer you to my other rant about stagnation and oil.

And, listen to yourself. "Good riddance, no, I don't use DOS" but "Ain't broke is the battle cry of those looking for real problems to solve."

You can solve real problems in DOS, it's usable, it ain't broke, there are even some decent games (doom) and windowing systems (win3.1 and others) for it.

But Linux is better. DOS ain't broke, but Linux is better. So maybe VFS ain't broke, but plugins would be better. I guess we'll only know if we let Reiser4 merge...

Quoted 6 lines
Quoted 1 line

But, there are some things Reiser does better and faster than ext3,

Yes, I've heard it is supposed to be faster on huge directories, and doesn't run out of inodes. And it is more efficient spacewise on small files. But then again, space is extremely cheap today...

Speed isn't. CPU is, but not disk speed. And packing stuff more efficiently, without actually compressing it, will give you some of that speed.

Also, space is not so cheap that I won't take 25% more.

Quoted 3 lines

And again, on a list around here I've seen several cries for help with completely hosed filesystems, all ReiserFS. No solution has ever come forth.

I haven't been counting, but I've seen a number of solutions fly around reiserfs-list for people with reiser4 problems.

Quoted 7 lines
Quoted 4 lines

even
if you don't count file-as-directory and other toys. There's nothing ext3 does better than Reiser, unless you count the compatibility with random bootloaders and low-level tools.

For me, those are quite critical...

There have been patches...

Quoted 24 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

Not everyone will want
to reformat at once, but as the reiser4 code matures and proves itself (even more than it already has),

Quoted 8 lines
Quoted 3 lines

I for one have seen mainly people with wild claims that it will make their machines much faster, and coming back later asking how they can recover their thrashed partitions...

You know how many I've had thrashed on Reiser4? Two. The first one was with a VERY early alpha/beta, and the second one was when I dropped a laptop and the disk failed.

OK. Know how many I thrashed with ext2/3? I remember 3, could have been as many as 5. One was due to a failed disk, another one because of DMA to a disk causing slow corruption. Another one I believe was due to a odd kernel compiled with a snapshot gcc a long time back, plus power loss at the wrong time. And that is from using ext2/3 since they were in early beta. On several machines at the same time, over years. I'd have to say that there isn't much of a difference.

Quoted 6 lines
Quoted 3 lines

And it does make certain things faster. For one thing, "emerge sync" on Gentoo is twice to four times as fast, and /usr/portage is 75% as big, as on ReiserFS (3).

That can't all be due to on-disk format.

It's not compression. What do you think it is?

To be fair, it's gotten a bit slower recently. Wish that repacker was done. I think it's actually gotten just a little slower than it ever was on v3, but v3 was awhile ago for me, and Gentoo has been growing a lot. /usr/portage is probably twice as big now as it was then.

If you want, I can do some unscientific benchmarks.

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 2 lines

the ext3 people may find themselves
wanting some of the more generic optimizations.

They'll filch them in due time, don't worry.

Quoted 6 lines
Quoted 3 lines

Duplication of effort. With plugins, we can optimize the upper layers of ALL filesystems, regardless of the lower layers, in such a way that it is optional.

Generic optimizations how, if they need VFS support?!

This was about a hypothetical ext3 format as a reiser4 storage plugin. I'm not sure how this ties into the VFS stuff.

Quoted 5 lines
Quoted 2 lines

I'm sure it's far easier to write a Reiser storage
plugin than a brand new FS.

Comparing apples and oranges tells you what?

That I'd rather have oranges?

A lot of what people like about ext3 is its stability and fairly universally accepted format. A lot of what people like about XFS is its stability and speed, mainly with large files. A lot of what people like about Reiser4 (as it is today) is its speed, with large and especially with small files.

Those are broad and somewhat uneducated statements, but I doubt most people care what FS they are using if the stability and performance is what they want. In that case, why have so much duplicated effort between different filesystems -- even between ISO and UDF and Reiser and XFS -- when most of what's really different is the on-disk format and the optimization?

So, in this hypothetical situation where ext3 is a reiser4 plugin, suddenly all the ext3 developers are trying to improve the speed and reliability of reiser4, which benefits both ext3 and reiser4, instead of just ext3.

Quoted 10 lines
Quoted 4 lines

Eventually, once competition is only based on storage format, we could end up with just one format. Just one filesystem! (except for fat/ntfs/iso/udf/network...) And in the open source world, sometimes a single product is a good thing.

Sorry, I don't think this will come to pass in our lifetime, if ever. There are different requirements, and the way to cater to them is different solutions. That has always been what Linux (as opposed to the propietary and even some open source systems) is all about...

egrep and fgrep are the same program. A lot of the code is shared, I'm sure. But you change some settings and they suddenly solve entirely different problems.

You don't need a wholly different solution if you can adapt an existing one. That's what plugins are all about -- adapting one solution to any number of requirements, so that people can focus on making that one solution better.

Quoted 1 line · as I
Quoted 1 line
Quoted 1 line
Quoted 1 line

But, I don't think that will realistically happen at all.

People are stubborn.

[...]

Quoted 7 lines
Quoted 5 lines
Quoted 5 lines
Quoted 1 line

and vfat people may decide they want cryptocompress,

I'm sure they don't, because it is mostly for Windows and assorted devices (pendrives, digital cameras, ...) compatibility.

Quoted 7 lines
Quoted 3 lines

I, for one, would like to use a pendrive and have certain files be encrypted transparently, only for use on Linux, but others be ready to transfer to a Windows box.

And that would surely break Windows compatibility (because you have to keep the data on what to encrypt one the filesystem itself). Besides, having

Aside from what someone else already said about this, why not just have support for accessing, say, a .gpg file as transparently decrypted? You don't even need file-as-directory, just create a file called foo which is really the decrypted version of foo.gpg. No need to change the format, just the filesystem.

Quoted 3 lines

pgp, or gpg, or crypt, or my own whacky encryption proggie do the job in /userland/, and not shoving into the kernel and so down the next user's throat, is in the end a largeish part of what Unix is all about for me.

Do you use ipv6? I don't. And it's not shoved down my throat, either, although it's in the kernel. I simply disable it, and you would (I'm sure) disable the crypto stuff.

Plus, as someone else said, it's much easier to do

$ vim /some/encrypted/file
than
$ gpg --decrypt /some/encrypted/file > /some/decrypted/file
$ vim /some/decrypted/file
$ gpg --encrypt /some/decrypted/file > /some/encrypted/file
$ shred /some/decrypted/file

Not to mention, shred doesn't work on modern filesystems, so you need to either patch vim (and any other program you might want to use), create an actual ramdisk, or deactivate swap and mount a tmpfs.

On doing stuff in userland: I am all for moving tons of stuff to userland, but not until someone does a microkernel right, if it's even possible. And even then, I'd probably still use Linux and argue for some stuff in the kernel, because Linux has developers. Developers. Developers. Developers.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQrzZOngHNmZLgCUhAQLk/w//dwfRaR5mtrznECpLMtpfNTm/n0EK2gqC ABYClsvcyKPzFtRTYnhZmBmVvAIueJpaReRmrR3QryESoeD6zftrxpuyvgcxxFRA BUIZcXKtqDF7ILmwCgSN0kB3ltD859q0E9ceJz5/lQZJwIJ2JN1huDM9xsOtw4sW 5HZEwSd6C46HArHD5fnnljppCgxV9ozVDfVzWs2GHU9r9cYZ+y9FU/kWIiuw78Ub fSSF7UqXd+QP0PWkkC7bAA9EtePT4Xd875WX7etgUmJRqVbf6erzPNq34b8V4r0r 6o2QFd5j4ZjpUvAeUzlMivSwFl10Yta+I9E3MHkYIm/86pX3GmPY5W24YfkDKsdS Y1odr3HYHt16QIJF3MPO3724PsSax+c9vmF7ANU+2QadtyQFrU0aasKxehTPnp9M PjShkWpWLGCG2YyesM3lq2HzydbljBraFUfzK9LKKoTqBk0jd6kcUrP8869NPo94 /stcyLSa/1gSAsSX+97Gs8CpQZ9JeJp7cMyDdrbR68h0xAg+UGK7tDez5j7VY9kc cSNo+T+W8OBiRPfz7Zg39ltHIeaPT/8THPKRmOLHWXK2duW+RmZBTDPK22Cwbv83 0f9RgRtm1q1FYG1q4gH373V/hjmvZIXGrN+/I5mU6P43iVrLuz/7MIWAGkQyXS3M nOCoZV7WiHY=
=XUC1
-----END PGP SIGNATURE-----

#227 lore raw

Re: reiser4 plugins

Quoted 3 lines · David Masover

But Linux is better. DOS ain't broke, but Linux is better. So maybe VFS ain't broke, but plugins would be better. I guess we'll only know if we let Reiser4 merge...

No, we'll only know if we merge something that does plugins at the VFS level in a well-designed way.

Quoted 2 lines

This was about a hypothetical ext3 format as a reiser4 storage plugin. I'm not sure how this ties into the VFS stuff.

Very poorly. There's only two interpretations of "ext3 as a reiser4 plugin" that make *any* sense. The first is that reiser4 is totally violating the VFS layer boundary, and the second is that reiser4 is trying to be an all-singing all-dancing wankfest. Later on, you say:

Quoted 5 lines

A lot of what people like about ext3 is its stability and fairly universally accepted format. A lot of what people like about XFS is its stability and speed, mainly with large files. A lot of what people like about Reiser4 (as it is today) is its speed, with large and especially with small files.

Now *think* for a moment - how does a hypothetical Reiser4 using ext3 format gain any speed advantage with small files, when the speed advantage is based on using a format other than ext3?

As I said, either it's violating the VFS boundary, or it's busy wanking.

The Reiser4 proponents would be well served to disavow that particular hypothetical example - I have yet to see *anything* that does more damage to the Reiser4 cause.

Quoted 4 lines

So, in this hypothetical situation where ext3 is a reiser4 plugin, suddenly all the ext3 developers are trying to improve the speed and reliability of reiser4, which benefits both ext3 and reiser4, instead of just ext3.

Or we can do what *should* be done, which is:

a) Put the crack pipe down.

b) Tell reiser4 to get its grubby little paws off the VFS if it ever intends to have a chance of being merged in mainline.

c) Have a *separate* project to improve the speed/reliability/function of the VFS layer, which is the only way that your vision of having the ext3 and reiser developers cooperating will ever happen.

Yes, the VFS could probably use an overhaul. But that *will* happen like this:

1) A patch is submitted and passes review to change the VFS.
2) If appropriate, a patch for reiser4 (if it gets merged) is also submitted (possibly by the same people) to be the first user of the new API/functionality.

There's a *reason* why we see patch streams that look like:

Patch 1/3: Add moby_foo_init function to nautical core. Patch 2/3: Modify white_whale driver to use moby_foo_init Patch 3/3: Modify captain_ahab driver to use moby_foo_init

Quoted 5 lines

Aside from what someone else already said about this, why not just have support for accessing, say, a .gpg file as transparently decrypted? You don't even need file-as-directory, just create a file called foo which is really the decrypted version of foo.gpg. No need to change the format, just the filesystem.

I don't think this is what they mean by "Linux gives you enough rope to shoot yourself in the foot with"...

Quoted 7 lines

Plus, as someone else said, it's much easier to do

$ vim /some/encrypted/file
than
$ gpg --decrypt /some/encrypted/file > /some/decrypted/file
$ vim /some/decrypted/file
$ gpg --encrypt /some/decrypted/file > /some/encrypted/file
$ shred /some/decrypted/file

You've totally failed to understand that the whole *point* of PGP is that 'vim /some/encrypted/file' *isnt* easy to do. A better example might be the various crypto-loop-ish variants or Microsoft's EFS, where the key management model is more tractable to this sort of automation.

#228 lore raw

Re: reiser4 plugins

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 10 lines · [email protected] wrote:
Quoted 3 lines · David Masover

But Linux is better. DOS ain't broke, but Linux is better. So maybe VFS ain't broke, but plugins would be better. I guess we'll only know if we let Reiser4 merge...

No, we'll only know if we merge something that does plugins at the VFS level in a well-designed way.

Which will never happen, the way you see it. (see below.)

Or maybe you could see this in action if you look at the *current* code, not the hypothetical future code where reiser4 plugins are removed from the FS, which is then merged, then plugins are added to VFS, then back
into reiser... (see below)

Quoted 8 lines
Quoted 2 lines

This was about a hypothetical ext3 format as a reiser4 storage plugin. I'm not sure how this ties into the VFS stuff.

Very poorly. There's only two interpretations of "ext3 as a reiser4 plugin" that make *any* sense. The first is that reiser4 is totally violating the VFS layer boundary, and the second is that reiser4 is trying to be an all-singing all-dancing wankfest. Later on, you say:

You'll have to be more specific and less insulting.

Actually, the way I'd like to see that is if we

Quoted 10 lines
Quoted 5 lines

A lot of what people like about ext3 is its stability and fairly universally accepted format. A lot of what people like about XFS is its stability and speed, mainly with large files. A lot of what people like about Reiser4 (as it is today) is its speed, with large and especially with small files.

Now *think* for a moment - how does a hypothetical Reiser4 using ext3 format gain any speed advantage with small files, when the speed advantage is based on using a format other than ext3?

It doesn't. It gains in other ways. Transparent cryptocompress, oddball other plugins, not to mention some speed optimizations that happen in RAM. If you do a ton of work with a dataset that stays in RAM, Reiser probably performs as well or better than a ramdisk, because changes that get overwritten while still in RAM never actually touch the disk. Reiser also doesn't fragment as quickly as ext3, and I don't think that has anything to do with its format.

Quoted 3 lines

The Reiser4 proponents would be well served to disavow that particular hypothetical example - I have yet to see *anything* that does more damage to the Reiser4 cause.

It is my example. I don't speak for Namesys. I don't work at Namesys.

If it does "damage", that is only in the eyes of those not paying attention.

Quoted 9 lines
Quoted 4 lines

So, in this hypothetical situation where ext3 is a reiser4 plugin, suddenly all the ext3 developers are trying to improve the speed and reliability of reiser4, which benefits both ext3 and reiser4, instead of just ext3.

Or we can do what *should* be done, which is:

a) Put the crack pipe down.

That was unneccesary. Come back when you can debate technical reasons, not imaginary personal ones.

Quoted 2 lines

b) Tell reiser4 to get its grubby little paws off the VFS if it ever intends to have a chance of being merged in mainline.

You are saying Reiser isn't in because it shouldn't touch VFS. Every single other person in this thread says Reiser isn't in because it *should* touch VFS.

Quoted 3 lines

c) Have a *separate* project to improve the speed/reliability/function of the VFS layer, which is the only way that your vision of having the ext3 and reiser developers cooperating will ever happen.

Why does it have to be a separate project if it is already *done* as part of Reiser4? Or is the name Reiser just cursed that way?

Quoted 5 lines

Yes, the VFS could probably use an overhaul. But that *will* happen like this:

1) A patch is submitted and passes review to change the VFS.
2) If appropriate, a patch for reiser4 (if it gets merged) is also submitted (possibly by the same people) to be the first user of the new API/functionality.

The FS that gets merged ahead of time without plugins would no longer be Reiser4. Go read the whitepaper, or tell me why I'm wrong, but even symlinks are implemented as plugins.

Quoted 5 lines

There's a *reason* why we see patch streams that look like:

Patch 1/3: Add moby_foo_init function to nautical core. Patch 2/3: Modify white_whale driver to use moby_foo_init Patch 3/3: Modify captain_ahab driver to use moby_foo_init

I'm not arguing that at all. But if you've got an entirely new driver, why not do:

Patch 1/2: Add white_whale driver, which also adds moby_foo_init to nautical core.
Patch 2/2: Modify captain_ahab driver to use moby_foo_init

Quoted 9 lines
Quoted 7 lines

Plus, as someone else said, it's much easier to do

$ vim /some/encrypted/file
than
$ gpg --decrypt /some/encrypted/file > /some/decrypted/file
$ vim /some/decrypted/file
$ gpg --encrypt /some/decrypted/file > /some/encrypted/file
$ shred /some/decrypted/file
Quoted 4 lines

You've totally failed to understand that the whole *point* of PGP is that 'vim /some/encrypted/file' *isnt* easy to do. A better example might be the various crypto-loop-ish variants or Microsoft's EFS, where the key management model is more tractable to this sort of automation.

Actually, plugins are just as easy or easier than crypto-loop or dm-crypt. And why shouldn't my crypto be easy? Most users are insecure in all kinds of ways because of that attitude -- security is HARD, so I won't do it. If security is transparent, just enter a password and go, then more people would do it. If more people are secure in more ways, there are less DDOS attacks and the world is a better place. -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr2jd3gHNmZLgCUhAQIeihAAgifgKxYw2FECNKLsspQ7sMOmBaT1jcev 5CJdDCiDjiMEMMcsh1h749tZX38YTjqL/w8VQQ18ys/2QqRERbjNEn/lGWLjO4Hs UwFFv/usVMv3RmRModH62P2j7QAxWhjTcfWYeIexdh20whMJLJA+BpIhnoKZ6Xjq WQMbdc9X/NW/lS09zkthWUju8JXtjTjD5rsMF3yQ47Scijec2i67ATsnzNxdXB5/ fZmjOfWKkahHh8qdO/IXX/6nvyEEzmVThr1EskCjUwoRYUSfb9bLQMkntjvEWSDh mDIBbwAUg88k3dM947uS55iV3QGoVkgDRvF0cpWxeaOEP+FcKmF9gPiaW0wm5jYw n60tw7P48FVchBrAPZTwvq5po78Evj3bB09NmS3i68in56rhRI3hTB7KNR/vIbS1 9dKvYuilV0C+thGD+sfE81qqFyP/O66sjvpxRi/YhjL8hU8N3x2f5tTIkKli7Y2C bZSxCiLiDRzKYAgCq0Rr+mrn/TZb5Ld5W8Ys062qF5l00M0I6cDO0sPiH72RexM7 PMM5tWNBDqVacUWxNYReULC5O7Q6c8FYTWcZz/zC1boLTeGpvlI8uVkbxP6K/grY j1u7vHbjgAqsztYhA8qmw3nl5vHtXCGHCRFQJ36g14Qt3atFcMgvfE5yphWyC1P9 hYZm8EWvMqc=
=GvIJ
-----END PGP SIGNATURE-----

#229 lore raw

Re: reiser4 plugins

Quoted 8 lines · David Masover
Quoted 3 lines

Now *think* for a moment - how does a hypothetical Reiser4 using ext3 format gain any speed advantage with small files, when the speed advantage is based on using a format other than ext3?

happen in RAM. If you do a ton of work with a dataset that stays in RAM, Reiser probably performs as well or better than a ramdisk, because changes that get overwritten while still in RAM never actually touch the disk.

At that point, since the actual buffer management is being done at the VFS level (see fs/buffer.c and friends) what you're really comparing is the speed of journalling metadata - at which point you need to be *very* careful to specify exactly what configuration you're talking about. If you don't believe me, investigate why mounting a filesystem with 'noatime,nodiratime' can make a dramatic difference totally independent of the underlying filesystem, but the actual amount of gain is dependent on format (hint - how far do the heads have to move to record 3 atime updates against 3 random inodes on an ext2, an ext3, and a VFAT filesystem, assuming no other disk activity), and why ext3 has 3 different modes data=ordered/writeback/journal.

Quoted 2 lines

Reiser also doesn't fragment as quickly as ext3, and I don't
think that has anything to do with its format.

Care to explain why it's not format-dependent?

Quoted 6 lines
Quoted 2 lines

b) Tell reiser4 to get its grubby little paws off the VFS if it ever intends to have a chance of being merged in mainline.

You are saying Reiser isn't in because it shouldn't touch VFS. Every single other person in this thread says Reiser isn't in because it *should* touch VFS.

Hmm.. let's see.. I said Reiser isn't in because it shouldn't be screwing with the VFS, and said stuff should be done separate from the Reiser4 filesystem.

Everybody else said that to achieve all the goals that Hans wants will require changes to the VFS, and the way reiser4 gets there isn't acceptable.

Seems like myself and everybody else are saying this needs to be factored into 2 pieces, a FS piece and a VFS piece, and moved forward separately.

Quoted 6 lines
Quoted 3 lines

c) Have a *separate* project to improve the speed/reliability/function of the VFS layer, which is the only way that your vision of having the ext3 and reiser developers cooperating will ever happen.

Why does it have to be a separate project if it is already *done* as part of Reiser4? Or is the name Reiser just cursed that way?

Because it's done *as a part of reiser4*, and not as a separately reviewed change to the VFS.

Quoted 3 lines

The FS that gets merged ahead of time without plugins would no longer be Reiser4. Go read the whitepaper, or tell me why I'm wrong, but even symlinks are implemented as plugins.

Which is another way of saying Reiser4 can't be merged in its present form.

Quoted 5 lines

I'm not arguing that at all. But if you've got an entirely new driver, why not do:

Patch 1/2: Add white_whale driver, which also adds moby_foo_init to nautical core.

You don't do this because the rule is "one patch, one logical change", and "which also" implies more than one change.

Quoted 3 lines

Actually, plugins are just as easy or easier than crypto-loop or dm-crypt. And why shouldn't my crypto be easy? Most users are insecure in all kinds of ways because of that attitude -- security is HARD, so I

There's a vast distinction between "easy for implementors" and "easy for users". Jaari Russo's loop-aes stuff does a wonderful job of being "easy for users" - just say "mount", answer the passphrase, and you're good to go. The underlying arguing about the crypto involved is complicated enough keep professional crypto jocks busy for years (is the watermark attack Jaari is concerned about a real threat? You tell me. ;)

Meanwhile, PGP was designed to be used in an environment where you could do
this: "Today's secret plans are AES256 encrypted. The key is the next key in your one-time-pad book, XOR'ed with your 128-bit secret key - do it in your head". (And yes, you can easily memorize a 16-digit hex number and learn to do an XOR with another 16-digit number, if failing to do so means you could end up dead).

This is inconvenient for the user, but intractable for an attacker to create a scenario where they can just 'vi /each/decrypted/file' ;)

Quoted 2 lines

won't do it. If security is transparent, just enter a password and go, then more people would do it.

"Just enter a password and go, then more people would do it".

Two words: "phishing e-mail".

Why does phishing work at all? Because it's simple for the user, and the user isn't aware of the totally busticated underlying security model of SMTP (namely, there isn't one) and the mostly busticated security model of most browsers (the misleading concept that if an SSL site is identified by the SSL cert, that this implies the site can be trusted, and other similar misfeatures).

#230 lore raw

Re: reiser4 plugins

Quoted 6 lines · [email protected] wrote:

Hmm.. let's see.. I said Reiser isn't in because it shouldn't be screwing with

the VFS, and said stuff should be done separate from the Reiser4 filesystem.

We don't touch a line of VFS code. We look like a normal fs at the interface.

Shall we send in a file labeled reiservfs.c containing the plugin layer? That will really test whether the Reiser name is cursed that way, yes?

There has been no response to the technical email asking for what exactly it is that is duplicative, and what exactly it is that ought to be changed in how the code works, as opposed to what file the code is
placed in, or who is considered its maintainer. There has been no
response to the questions about whether the difference between class and instance makes our layer non-duplicative.

Perhaps no response was possible?

Hans

#231 lore raw

Re: reiser4 plugins

Quoted 10 lines · Hans Reiser

There has been no response to the technical email asking for what exactly it is that is duplicative, and what exactly it is that ought to be changed in how the code works, as opposed to what file the code is
placed in, or who is considered its maintainer. There has been no
response to the questions about whether the difference between class and instance makes our layer non-duplicative.

Perhaps no response was possible?

Hans,

the l-k community have asked YOU may times. any lack of response isn't because of the kernel cabal .. its because YOU are refusing to entertain any notion that what Reiser4 has implemented is unpalatable to the kernel community.

you can threaten all you want to take your code and move it to BSD. or fork the kernel. whatever.
but if you want to get your work into the mainline kernel, you need to provide answers to the question that keeps being asked of you - and which you patently keep ignoring time & time again.

in short, as per Message-ID: <[email protected]>:
posting to l-k on "why Reiser4 cannot use VFS infrastructure for [crypto,compression,blahblah] plugins" - ideally, for each plugin.

or again, in Message-ID: <[email protected]>:
[..] but instead just understand that this is the framework that you
have to
work in to get it into the mainline kernel.
if you don't want to go down that path, you're free to do so. its open source, you can keep your own linux-kernel fork. but if you want to get your code into mainline, i don't think its so much a case of l-k folks telling you how to make your code work under VFS. the onus is on you to say WHY your code and plugins could never be put into VFS.

or further back in Message-ID: <[email protected]>
you know that VFS is the mechanism in Linux. you know (i hope..) how it works. it isn't so hard to see how many of the Reiser4 "plug-ins" could be tied into VFS calls. OR, if they cannot TODAY, propose how VFS _COULD_ be made to do this.

NB. it doesn't matter what David thinks. this is linux-kernel, not linux-users.

cheers,

lincoln.

Quoted 2 lines
#232 lore raw

Re: reiser4 plugins

Quoted 4 lines · Lincoln Dale

the l-k community have asked YOU may times. any lack of response isn't because of the kernel cabal .. its because YOU are refusing to entertain any notion that what Reiser4 has implemented is unpalatable to the kernel community.

A lot of this is based on misconceptions, for example in recent times reiser4 is faulted for layering violations.. But it doesn't have them, it neither duplicates nor modifies the VFS.

It has also been requested that reiser4 be changed to move some of it's operations above the VFS... not only would that not make sense for the currently provided functions, but merging was put off previously because of changes to the VFS.... now that it doesn't change the VFS we are asking hans to push it off until it does??

It's a filesysem for gods sake. Hans and his team have worked hard to minimize its impact and they are still willing to accept more guidance, even if their patience has started to run a little thin. The acceptance of reiser4 into the mainline shouldn't be any larger deal than any other filesystem, but yet it is...

#233 lore raw

Re: reiser4 plugins

Quoted 14 lines · Gregory Maxwell
Quoted 6 lines · Lincoln Dale

the l-k community have asked YOU may times. any lack of response isn't because of the kernel cabal .. its because YOU are refusing to entertain any notion that what Reiser4 has implemented is unpalatable to the kernel community.

A lot of this is based on misconceptions, for example in recent times reiser4 is faulted for layering violations.. But it doesn't have them, it neither duplicates nor modifies the VFS.

Quoted 7 lines

It has also been requested that reiser4 be changed to move some of it's operations above the VFS... not only would that not make sense for the currently provided functions, but merging was put off previously because of changes to the VFS.... now that it doesn't change the VFS we are asking hans to push it off until it does??

<sigh>

it has NEVER been a case of Reiser4 not being merged because "it required changes to VFS".

the whole point of VFS is to provide a standard API for data to/from individual filesystems.
over the course of history, VFS itself hasn't been a static thing - it has had to adapt and change as a result of the needs of filesystems.
but it hasn't ever been a case of individual filesystems doing 'proprietary' things (i.e. there isn't a sys_ext3() system call) - where it has made sense to do things at a VFS layer, VFS itself has been adapted to handle those things.

a semi-recent example of this is extended attributes.
it is with some irony that on my desktop i make use of the excellent open-source desktop search tool 'beagle'.
it, however, uses extended attributes for storing things - and Reiser4's EAs are incompatible with the "standard" EAs such that Reiser4 is incompatible with beagle.

this is the WHOLE point of standardization .. i don't think its that Reiser4's EAs offer any more or less capabilities than standard EAs - BUT they haven't used the standard mechanisms available for implementing them, such for Beagle to work on Reiser4, there now needs to be logic added to Beagle to do so.

lets take this a step further. what about compression? do we accept that each filesystem can implement its own proprietary compression via its own API - and now we need individual user-space tools to understand each of these APIs? how about encryption?
... and so-on.
suddenly every user app out there needs to have specialized knowledge of each type of filesystem.

Hans should be applauded for the 'plug-in' concept and showing how it can be used. however, from an implementation stand-point, it really shouldn't come as any great surprise that numerous kernel developers are pushing back saying 'layering violation' and "why can't this be done at the VFS layer".

none of this is rocket-science. its just plain common sense.

Quoted 4 lines

It's a filesysem for gods sake. Hans and his team have worked hard to minimize its impact and they are still willing to accept more guidance,

i don't see any acceptance at this point. simply lots of hot air that smells like marketing & PR.

cheers,

lincoln.

#234 lore raw

Re: reiser4 plugins

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lincoln Dale wrote:

[...]

Quoted 2 lines

this is the WHOLE point of standardization .. i don't think its that Reiser4's EAs offer any more or less capabilities than standard EAs -

They do. Reiser4's EAs can look like any other object -- files, folders, symlinks, whatever. This is important, especially for transparency.

For one thing, can I access a Beagle search as a folder?

Quoted 3 lines

BUT they haven't used the standard mechanisms available for implementing them, such for Beagle to work on Reiser4, there now needs to be logic added to Beagle to do so.

Well, ideally, I'd like to see people stop bickering, come up with something better than sys_reiser4, add an emulation layer for xattrs and mark them obsolete.

But I don't speak for Namesys.

Quoted 3 lines

lets take this a step further. what about compression? do we accept that each filesystem can implement its own proprietary compression via its own API - and now we need individual user-space tools to understand

No, that's the beauty of these "EAs" in Reiser4. The API is standard write(2) commands. sys_reiser4 supposedly implements an interface to make this scale better, but otherwise have the same semantics. And who said anything about proprietary compression? I think we were planning on the kernel's zlib, though we might have been planning to make it a bit more seekable...

Quoted 1 line

each of these APIs?

So, the API becomes something like:

cat crypto/inflated/foo # transparently decompressed
cat crypto/raw/foo.gz # raw, gzip-compressed

Another possibility, if you like file-as-a-directory:

cat foo.gz # raw
cat foo.gz/inflated # decompressed

One could easily imagine things like these two potentially equivalent commands:

cp foo bar.zip/
zip bar foo

The whole point is to have less userland tools, not more. I'm not saying we move zip into the kernel, just that the user now has one less command to remember.

But, back to reality. file-as-directory probably won't happen, at least not for awhile, so imagine more along the lines of my first example.

Quoted 1 line

how about encryption?

About the same, only now you have a key file that you write to in order to unlock the decrypted files.

Quoted 3 lines

... and so-on.
suddenly every user app out there needs to have specialized knowledge of each type of filesystem.

Not really.

More like, every app that cares to has generalized knowledge, if that.

Quoted 1 line

none of this is rocket-science. its just plain common sense.

I could say the same of my stuff, but lots of people seem to disagree with me, or at least fail to see it. I guess I can say the same of your comments.

Quoted 6 lines
Quoted 4 lines

It's a filesysem for gods sake. Hans and his team have worked hard to minimize its impact and they are still willing to accept more guidance,

i don't see any acceptance at this point. simply lots of hot air that smells like marketing & PR.

They do keep asking for specifically what they need to do to put this stuff in VFS. Or am I wrong?

Or maybe it should be obvious to them?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr5dtngHNmZLgCUhAQIBGxAAiLK6EyHnLRhEA+rUIDCwacM4K89wlE7X +dcw3xv3Pc9tZZqVVAd7Y27whEzjmNOwfGkvPkzPk/ATQditnyt+7xHcuXpqORNU j7zHc5zS8MGDRU8Re4MXTO6jCXDgtTwQHjcdg4i8KYWLMPT7LpO+DHY/mZyQEgpD kZGE4WJePA+aNlHAzySW9u/atnwp5hSRvmfuF4zzN8ng5tf8SSMvbfoCyjYSue8l N6jvcGnt+yItmbHVaij0IdHUw1/9/u6b3Q0Ut39NBk8fUKXJcASHmKtjwLTAoWW+ hiYVhLdZQGkWo2d6XdzdNY2OgE3kWVnLBqrOuTo7zCjMojvWIrEGE/x3Yh/E6Hs8 cAPVRebG5yUQBxJk1lcDeJOBozutIpCVyTzwBnKU1nz3KArqanU51oT++3cTjVha 1tBnaLS4RLcdy8UD1ewS+VHj61VSlcBjv2abCrYw4DC0anUFrYUSciNjx3tdYJRx o7l/pEn7UYPpGaXgHyBdVDIRlRNdOoTRZp3aIY2Z2v6/jyi3TeufMUjGtpuQHl2k BuYm7tV4l1Ec/QZLM+PbAyVU9qqlz9BuHlI1U7z1p3gYeAzz0guAeDHfi1l95sUn l+bFCOfmXi3qRAxVZyidczqeOtGtCed3nIUH+1z+siuFzH3jecjzUpGWKcGxzVlc jkUS+tlihfg=
=C4UP
-----END PGP SIGNATURE-----

#235 lore raw

Re: reiser4 plugins

Quoted 9 lines · David Masover

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lincoln Dale wrote:

[...]

Quoted 10 lines
Quoted 4 lines

this is the WHOLE point of standardization .. i don't think its that Reiser4's EAs offer any more or less capabilities than standard EAs -

They do. Reiser4's EAs can look like any other object -- files, folders, symlinks, whatever. This is important, especially for transparency.

it was accepted not so long ago that 'file-as-directory' and 'EA' are two different things, predominantly because existing tools and apps won't necessarily "do the right thing<tm>".

this has been discussed to death previously. oh what a surprise. http://lwn.net/Articles/100271/

cheers,

lincoln.

cheers,

lincoln.

#236 lore raw

Re: reiser4 plugins

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 18 lines · Lincoln Dale
Quoted 14 lines · David Masover

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lincoln Dale wrote:

[...]

Quoted 3 lines

this is the WHOLE point of standardization .. i don't think its that Reiser4's EAs offer any more or less capabilities than standard EAs -

Quoted 8 lines
Quoted 5 lines

They do. Reiser4's EAs can look like any other object -- files, folders, symlinks, whatever. This is important, especially for transparency.

it was accepted not so long ago that 'file-as-directory' and 'EA' are two different things, predominantly because existing tools and apps won't necessarily "do the right thing<tm>".

They are different, but not quite -- what's the word -- discrete? Rather, file-as-directory is one logical conclusion of EA, and one possible representation of EA.

Anyway, Reiser's EAs currently (I think) could appear as files, but not necessarily the file-as-directory madness we were talking about then. They could also be accessed by openat. And they can also be hierarchical, which I would think makes them cleaner and faster.

Quoted 2 lines

this has been discussed to death previously. oh what a surprise.
http://lwn.net/Articles/100271/

I was there. That was fun to watch, actually, once it got over my head.

The three points he lists are only relevent if you try to do file-as-directory. Right now, I like the idea of file attributes as showing up in separate magic directories, or just creating magic directories first and filling them with files. Symlinks could help clean up messiness, and allow you to do things like encrypt /etc/passwd
- -- symlink to /etc/crypto/passwd/decrypted or something. Yes, I know about shadow, and there are still some paranoid reasons to encrypt passwd.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr521XgHNmZLgCUhAQKSSA/9HK8JiG6wW40jQDUXsyKPz+l9UvoaNuLs hqIXo3sdQj1CWAuwa1BKY16w91tJBN3uX75jLgwR36ix4A1jXQ5v1sRjvfkLKR/7 3a51v9UsRAhAisiqWFGWYTrbXgAh+S5B51xHuXv3qTs/wqC8sxuJtAHwuCldPaBr cR1WzPtGvgd+ESqx5avllZIfNCy9tcH3P5fUsaIFCCm30E6u9PVO6xBOHylFWZKS Pywv+wGUTxbgCFSmLG07/zhwVF94fAHPIWXQXQGmhPrGf3Wtt7VTMiRpkpyONyso eoY1hFiwyh2YMrIPxdL0Uo+mcDvErWFFZyTcCGqIMp6x+QSe0Ww9Ie90afZPvvS0 Q4DmdST2JEHEinal5MCiqr3S83wanv3+h9ywTCEkTcl3mEK1iwtc4dwmvRNVHfkx f34CAxM1rBfu++kd3xgL+Kb/ao4LOCDhVL3XY6pNDglX6Y+Kk5dRSJjOm4kAU2fB j66uGOUZOiCCzSMLvVrCAPnNXmUavLKUXDKXKL2gTCYM6TL+7RkAXEMrqu5YxdZM ihIbfmGc7FzItbQCDZhhozG51IMkLCotU9U9CaotnhfUaosibmwEnmeY8FWY75ba MOs+VH3UJUZObRBwmBHSX4pwg5sDhSjyqPR05M26A5xz8nGh4kukD/E6QZYCB9EP lVdDus4iJUM=
=hsOJ
-----END PGP SIGNATURE-----

#237 lore raw

Re: reiser4 plugins

In reply to David Masover
Quoted 8 lines · David Masover
Quoted 2 lines · Lincoln Dale

this is the WHOLE point of standardization .. i don't think its that Reiser4's EAs offer any more or less capabilities than standard EAs -

They do. Reiser4's EAs can look like any other object -- files, folders, symlinks, whatever. This is important, especially for transparency.

No, you want them to look like the same objects that {get|set}xattr() manage currently. You don't want programs to have to guess what an EA looks like this week, with this user's combination of plugins that's different from everybody else's.

Quoted 11 lines
Quoted 3 lines

lets take this a step further. what about compression? do we accept that each filesystem can implement its own proprietary compression via its own API - and now we need individual user-space tools to understand

No, that's the beauty of these "EAs" in Reiser4. The API is standard write(2) commands. sys_reiser4 supposedly implements an interface to make this scale better, but otherwise have the same semantics. And who said anything about proprietary compression? I think we were planning on the kernel's zlib, though we might have been planning to make it a bit more seekable...

Quoted 6 lines
Quoted 1 line

each of these APIs?

So, the API becomes something like:

cat crypto/inflated/foo # transparently decompressed
cat crypto/raw/foo.gz # raw, gzip-compressed

And 'cat crypto/raw/foo' or 'crypto/inflated/foo.gz' gets you what, exactly?

Now throw some .bz2 and .zip files into the mix... ;)

Quoted 10 lines

Another possibility, if you like file-as-a-directory:

cat foo.gz # raw
cat foo.gz/inflated # decompressed

One could easily imagine things like these two potentially equivalent commands:

cp foo bar.zip/
zip bar foo

Unless of course the user had done 'mkdir sorted.by.city.zip' to make a directory of files containing data sorted by USPS Zip code.

And what happens if the user has a file 'bar' that's not a ZIP file, and a directory 'bar.zip' isn't a view into 'bar'?

Most of the time, if I have a file 'linux-2.6.12.tar.bz2' and a directory 'linux-2.6.12', what is under the directory is *NOT* the same data as what's in the .bz2 - I've done 'make oldconfig' and a few builds and some variable amount of patching, usually with rejects, and I *don't* want that .bz2 being updated during all this (hint - what's my next command after 'rm -rf linux-2.6.12' likely to be, and why, and what expectations do I have when I do it?)

You want to think this sort of thing through *really* thoroughly, because there's a *lot* of things, both users and programs, that have expectations about The Way Things Work.

Quoted 3 lines

The whole point is to have less userland tools, not more. I'm not saying we move zip into the kernel, just that the user now has one less command to remember.

But now instead of having to remember the one meme "I can manage any compressed-archive format that's stored in a file, and put other files in it, and all I need is the appropriate userspace tool", they have to remember "the cp trick works for .zip and .tar, but I'll get a "not a directory" error if I try it with a .hqx file, and that other file format may or may not work, because I can't remember if this kernel has a working out-of-tree module for this kernel...."

#238 lore raw

Re: reiser4 plugins

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 14 lines · [email protected] wrote:
Quoted 9 lines · David Masover
Quoted 2 lines · Lincoln Dale

this is the WHOLE point of standardization .. i don't think its that Reiser4's EAs offer any more or less capabilities than standard EAs -

They do. Reiser4's EAs can look like any other object -- files, folders, symlinks, whatever. This is important, especially for transparency.

Quoted 4 lines

No, you want them to look like the same objects that {get|set}xattr() manage currently. You don't want programs to have to guess what an EA looks like this week, with this user's combination of plugins that's different from everybody else's.

"Plugins" is a bad word. This user's combination of plugins is most likely identical to other users', it's just which ones are enabled, and which aren't? If they are all included, I assume they play nice.

And just because they are called "plugins" doesn't mean the EA looks different every week.

I cannot stress this enough. In Reiser4, EVERYTHING is some sort of plugin. We now have a standard disk format plugin, and a standard set of plugins providing UNIX filesystem-like things (vanilla files, directories, symlinks, device nodes, named pipes...) which has been stable for some time.

I see no problems with providing at least as consistent an EA interface as any other FS, only more extensible because it's not flat.

Quoted 16 lines
Quoted 16 lines
Quoted 3 lines

lets take this a step further. what about compression? do we accept that each filesystem can implement its own proprietary compression via its own API - and now we need individual user-space tools to understand

No, that's the beauty of these "EAs" in Reiser4. The API is standard write(2) commands. sys_reiser4 supposedly implements an interface to make this scale better, but otherwise have the same semantics. And who said anything about proprietary compression? I think we were planning on the kernel's zlib, though we might have been planning to make it a bit more seekable...

Quoted 1 line

each of these APIs?

So, the API becomes something like:

Quoted 7 lines
Quoted 2 lines

cat crypto/inflated/foo # transparently decompressed
cat crypto/raw/foo.gz # raw, gzip-compressed

And 'cat crypto/raw/foo' or 'crypto/inflated/foo.gz' gets you what, exactly?

Now throw some .bz2 and .zip files into the mix... ;)

Interface is the same. Only, zip files aren't just compression, so maybe the interface changes a little there.

Point is, now you have a standard interface for any program to access any simple lossless compression, transparently.

Quoted 8 lines
Quoted 8 lines

Another possibility, if you like file-as-a-directory:

cat foo.gz # raw
cat foo.gz/inflated # decompressed

One could easily imagine things like these two potentially equivalent commands:

Quoted 6 lines
Quoted 2 lines

cp foo bar.zip/
zip bar foo

Unless of course the user had done 'mkdir sorted.by.city.zip' to make a directory of files containing data sorted by USPS Zip code.

What's this got to do with anything?

Quoted 2 lines

And what happens if the user has a file 'bar' that's not a ZIP file, and a directory 'bar.zip' isn't a view into 'bar'?

In file-as-a-directory (which is probably NOT happening soon), bar.zip is both the actual zipfile and the view inside, depending on whether you try to open() it directly or peek inside it as a directory.

I've used that interface for some simpler things like file permissions.
I could do things like "echo 0 > some_file/metas/uid". The interface
is gone for now, because it broke some things, I'm told, but nothing serious for me while I was using it. I don't think the problem is insurmountable.

However, let's not discuss this now. I do NOT want to start another "silent semantic changes with reiser4" thread. File-as-directory is not happening this time, so don't worry about it -- this time.

Quoted 7 lines

Most of the time, if I have a file 'linux-2.6.12.tar.bz2' and a directory 'linux-2.6.12', what is under the directory is *NOT* the same data as what's in the .bz2 - I've done 'make oldconfig' and a few builds and some variable amount of patching, usually with rejects, and I *don't* want that .bz2 being updated during all this (hint - what's my next command after 'rm -rf linux-2.6.12' likely to be, and why, and what expectations do I have when I do it?)

You're misunderstanding. man zip.

$ zip bar foo
creates/modifies a file named "bar.zip", not "bar", which contains the
file "foo".
Quoted 3 lines

You want to think this sort of thing through *really* thoroughly, because there's a *lot* of things, both users and programs, that have expectations about The Way Things Work.

Or, I can avoid those issues altogether, and simply delegate this kind of stuff to user-created-but-magic directories. For instance, I could have a directory called "/foo" which contains encrypted files, and "/foo/decrypted" which has transparently decrypted representations of them.

Quoted 5 lines
Quoted 3 lines

The whole point is to have less userland tools, not more. I'm not saying we move zip into the kernel, just that the user now has one less command to remember.

Quoted 7 lines

But now instead of having to remember the one meme "I can manage any compressed-archive format that's stored in a file, and put other files in it, and all I need is the appropriate userspace tool", they have to remember "the cp trick works for .zip and .tar, but I'll get a "not a directory" error if I try it with a .hqx file, and that other file format may or may not work, because I can't remember if this kernel has a working out-of-tree module for this kernel...."

True, for now. But for me, it's not hard at all to try "cp" first, and after it fails, Google and install the required tool. After all, most users already have to do that with GUI interfaces, this just takes the GUI way of doing things (like in file-roller) and makes it perform better and work with all programs. Not necessarily all formats, but all programs.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr8IzngHNmZLgCUhAQKS9A/+K+Nkq0sLynA0ltiA7o+BnA4cRSLYnu0t LdVI+yvYMUocvZt1HcIuU1LN2y3//Xm7hh7Lsyf3moNCIFMkDUN/xuzUcmfpUs5W NpyYPS4M3h8bX7nAMI9SmJ2KPoQNJxiEMToi5cYkQvafAeAlR2xw+z9nQ7Cv2PAi PCKROYSbZ1MdyCVLDPYkIpk9Z9EnOZ7zE+nbMaippqAnwfoqMoe8OMBKt1qxBROS tSPn4h1+JQZTs5OY18Cg/km7J6SM3SmHEWtsicil9/AVMFoOCpUXkeuiBLgVZ7Vk hh79SLjmuAOgU5hTBhEwvtEjB5uGqbjRtp+quIWNO4tmHqPuSv+TP0YaOgMTqbf7 9Cr+iq/YrWMDCA3fYrnq/TiSCNK/YHGNlQG4rZtnKDXUFUZRuItMhbRaiYYfv/Cl wQjBrknKci4xFNB9gIOoykS9Vs3JDdZc8oj26rFhHGqkGE+xo6b0z1EaZMvBfes9 L+VqU6B7YnyDWnhDbK6TiLZ/Qg83SigTyoLORVIDzyo3k2SVA8kmy8NgdA9NlmIT ko/g0+7zHtu64LuEtQEdL/FULnI/YcMtxQDjYuhCqtUo/x4y3hi5685P00iZDfdp RpuZnOT/esiw/RLashgGRK3xsVik9R+WX2psrEXp7HvNYvr5X4BpFovyVYeTxSnj jV201bzTv6A=
=z+xB
-----END PGP SIGNATURE-----

#239 lore raw

Re: reiser4 plugins

Quoted 3 lines · David Masover

"Plugins" is a bad word. This user's combination of plugins is most likely identical to other users', it's just which ones are enabled, and which aren't? If they are all included, I assume they play nice.

Which ones are enabled. Exactly.

Quoted 2 lines

And just because they are called "plugins" doesn't mean the EA looks different every week.

They do if the one enabled this week is "make EAs look like symlinks", and last week's was "make EAs look like folders".

(Don't blame me, *you're* the one that said "EAs can look like any other object"..)

Quoted 1 line
Quoted 1 line

And 'cat crypto/raw/foo' or 'crypto/inflated/foo.gz' gets you what, exactly

?

Quoted 5 lines
Quoted 2 lines

Now throw some .bz2 and .zip files into the mix... ;)

Interface is the same. Only, zip files aren't just compression, so maybe the interface changes a little there.

Right. So please explain what crypto/raw/foo and crypto/inflated/foo.gz give you.

Quoted 11 lines

Point is, now you have a standard interface for any program to access any simple lossless compression, transparently.

Quoted 8 lines
Quoted 8 lines

Another possibility, if you like file-as-a-directory:

cat foo.gz # raw
cat foo.gz/inflated # decompressed

One could easily imagine things like these two potentially equivalent commands:

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

cp foo bar.zip/
zip bar foo

Unless of course the user had done 'mkdir sorted.by.city.zip' to make a directory of files containing data sorted by USPS Zip code.

What's this got to do with anything?

It's got a *LOT* to do with it if I created a *DIRECTORY*, to use *AS A DIRECTORY*, the way Unix-style systems have done for 3 decades, and suddenly my system is running like a pig because the kernel decided that it's a .zip file.

Quoted 6 lines
Quoted 2 lines

And what happens if the user has a file 'bar' that's not a ZIP file, and a directory 'bar.zip' isn't a view into 'bar'?

In file-as-a-directory (which is probably NOT happening soon), bar.zip is both the actual zipfile and the view inside, depending on whether you try to open() it directly or peek inside it as a directory.

Ahem. "bar.zip' is a *DIRECTORY*. I said 'mkdir bar.zip' - why is it not acting like a directory?

Quoted 3 lines

However, let's not discuss this now. I do NOT want to start another "silent semantic changes with reiser4" thread. File-as-directory is not happening this time, so don't worry about it -- this time.

Fish or cut bait. You are the one who started handwaving the 'file-as-directory'. If you don't want it discussed, don't mention it.

Quoted 8 lines
Quoted 7 lines

Most of the time, if I have a file 'linux-2.6.12.tar.bz2' and a directory 'linux-2.6.12', what is under the directory is *NOT* the same data as what's in the .bz2 - I've done 'make oldconfig' and a few builds and some variable amount of patching, usually with rejects, and I *don't* want that .bz2 being updated during all this (hint - what's my next command after 'rm -rf linux-2.6.12' likely to be, and why, and what expectations do I have when I do it?)

Quoted 4 lines

You're misunderstanding. man zip.

$ zip bar foo
creates/modifies a file named "bar.zip", not "bar", which contains the
file "foo".

No. *YOU* are misunderstanding. I have a directory 'linux-2.6.12', and I have a file 'linux-2.6.12.tar.bz2', and I do *NOT* want directory operations to be silently converted into "let's scribble into the middle of this tar file and then compress it". (Hint - work out how long a kernel 'make' would take if you were doing it inside a .tar.bz2).

Quoted 8 lines
Quoted 3 lines

You want to think this sort of thing through *really* thoroughly, because there's a *lot* of things, both users and programs, that have expectations about The Way Things Work.

Or, I can avoid those issues altogether, and simply delegate this kind of stuff to user-created-but-magic directories. For instance, I could have a directory called "/foo" which contains encrypted files, and "/foo/decrypted" which has transparently decrypted representations of them.

So rather than everything working in a funky manner, a program gets to guess how funky, and in what direction, a given magical directory is....

#240 lore raw

Re: reiser4 plugins

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 9 lines · [email protected] wrote:
Quoted 3 lines · David Masover

"Plugins" is a bad word. This user's combination of plugins is most likely identical to other users', it's just which ones are enabled, and which aren't? If they are all included, I assume they play nice.

Which ones are enabled. Exactly.

I doubt there will be duplicate plugins, once things settle down. So you just have the program demand certain ones to be enabled.

Quoted 6 lines
Quoted 2 lines

And just because they are called "plugins" doesn't mean the EA looks different every week.

They do if the one enabled this week is "make EAs look like symlinks", and last week's was "make EAs look like folders".

We could just as easily do that with other extended attributes. Sure, the details would be different -- maybe we just randomly add a XA onto the beginning of every string.

Quoted 1 line

(Don't blame me, *you're* the one that said "EAs can look like any other object"..)

Doesn't mean we can't guarentee a certain kind in a certain configuration will always be available for a certain plugin, once it's been accepted.

Quoted 7 lines
Quoted 4 lines
Quoted 4 lines

And 'cat crypto/raw/foo' or 'crypto/inflated/foo.gz' gets you what, exactly

?

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

Now throw some .bz2 and .zip files into the mix... ;)

Interface is the same. Only, zip files aren't just compression, so maybe the interface changes a little there.

Right. So please explain what crypto/raw/foo and crypto/inflated/foo.gz give you.

In that example (shouldn't have used the name "crypto", but oh well), it should be crypto/raw/foo.gz and crypto/inflated/foo -- where foo.gz is the gzip'ed file and foo is the transparently compressed/decompressed file. Basically, these are equivalent:

$ zcat crypto/raw/foo.gz
$ cat crypto/inflated/foo
Quoted 19 lines
Quoted 19 lines

Point is, now you have a standard interface for any program to access any simple lossless compression, transparently.

Quoted 14 lines
Quoted 10 lines

Another possibility, if you like file-as-a-directory:

cat foo.gz # raw
cat foo.gz/inflated # decompressed

One could easily imagine things like these two potentially equivalent commands:

cp foo bar.zip/
zip bar foo

Unless of course the user had done 'mkdir sorted.by.city.zip' to make a directory of files containing data sorted by USPS Zip code.

Quoted 6 lines
Quoted 1 line

What's this got to do with anything?

It's got a *LOT* to do with it if I created a *DIRECTORY*, to use *AS A DIRECTORY*, the way Unix-style systems have done for 3 decades, and suddenly my system is running like a pig because the kernel decided that it's a .zip file.

The kernel does not decide that. You do. And it doesn't automatically decide that every time you create a file. You have to use some interface to trigger the plugins.

Originally, this was file-as-a-directory. Now? I'm not sure, I guess we could use a separate delimiter. Something like:

foo # file
...foo # directory containing xattrs of file, list of plugins used.

foo/ # directory
foo/... # directory containing xattrs of file, list of plugins used.

I guess I need a new name for this approach. That's three possible ways of doing this?

Quoted 10 lines
Quoted 6 lines
Quoted 2 lines

And what happens if the user has a file 'bar' that's not a ZIP file, and a directory 'bar.zip' isn't a view into 'bar'?

In file-as-a-directory (which is probably NOT happening soon), bar.zip is both the actual zipfile and the view inside, depending on whether you try to open() it directly or peek inside it as a directory.

Ahem. "bar.zip' is a *DIRECTORY*. I said 'mkdir bar.zip' - why is it not acting like a directory?

If you said "mkdir", it would act like a directory.

More likely than foo.zip/bar would be foo.zip/.../contents/bar. Which would also work for tarballs. But would not automatically compress anything you didn't want it to.

Quoted 7 lines
Quoted 3 lines

However, let's not discuss this now. I do NOT want to start another "silent semantic changes with reiser4" thread. File-as-directory is not happening this time, so don't worry about it -- this time.

Fish or cut bait. You are the one who started handwaving the 'file-as-directory'. If you don't want it discussed, don't mention it.

I do want it discussed. I'm not sure it's a good idea now. But looks like you got me...

Quoted 14 lines
Quoted 8 lines
Quoted 7 lines

Most of the time, if I have a file 'linux-2.6.12.tar.bz2' and a directory 'linux-2.6.12', what is under the directory is *NOT* the same data as what's in the .bz2 - I've done 'make oldconfig' and a few builds and some variable amount of patching, usually with rejects, and I *don't* want that .bz2 being updated during all this (hint - what's my next command after 'rm -rf linux-2.6.12' likely to be, and why, and what expectations do I have when I do it?)

Quoted 4 lines

You're misunderstanding. man zip.

$ zip bar foo
creates/modifies a file named "bar.zip", not "bar", which contains the
file "foo".
Quoted 5 lines

No. *YOU* are misunderstanding. I have a directory 'linux-2.6.12', and I have a file 'linux-2.6.12.tar.bz2', and I do *NOT* want directory operations to be silently converted into "let's scribble into the middle of this tar file and then compress it". (Hint - work out how long a kernel 'make' would take if you were doing it inside a .tar.bz2).

I remember discussing that, actually. It wouldn't automatically do this if you didn't want it to, but it would be nice if, say, it was something truly seekable like linux-2.6.12.zip, and linux-2.6.12 was a user-created symlink to linux-2.6.12.zip/.../contents, and we had a nice caching system...

This is nice because then you get exactly the same performance during "make" as you would with "unzip && make", only better, because files you don't ever use (lots of arch, for instance) are not unpacked.

This is all something that's been discussed before. I think the consensus was that it sounded cool, but not useful enough by itself to justify file-as-a-directory.

You can always go re-read "Silent Semantic Changes..."

Quoted 4 lines
Quoted 4 lines
Quoted 3 lines

You want to think this sort of thing through *really* thoroughly, because there's a *lot* of things, both users and programs, that have expectations about The Way Things Work.

Quoted 8 lines
Quoted 4 lines

Or, I can avoid those issues altogether, and simply delegate this kind of stuff to user-created-but-magic directories. For instance, I could have a directory called "/foo" which contains encrypted files, and "/foo/decrypted" which has transparently decrypted representations of them.

So rather than everything working in a funky manner, a program gets to guess how funky, and in what direction, a given magical directory is....

It doesn't have to guess, it can know if it needs to. Although this probably wreaks havok on traditional backup systems. You'd have to implement a somewhat new backup system, but it's not impossible, and certainly doesn't require that your new backup system know about every plugin -- just make sure plugins know about backup.

Of course, file-as-directory avoids all that -- you just have to patch programs to know that if foo/ exists or foo responds to opendir, foo is not necessarily a directory. You probably just drop the trailing / and use stat().

Both (all three?) appraches have cans-of-worms and workarounds. But I think both (all three?) are doable. I know user-created-but-magic is doable.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr8txHgHNmZLgCUhAQKOxA/+KP1lmUgmrbEH+kPKriSFF3qFfFBgbeJh iebHVaOY4AMa99a3yHy8iA5ZyE5N2b3CEueoq44ygYs5gqeft1IF14SJweNk/6or IgGt+OsAkqDMdbbpKZdDGo8lZK0bJq/Ni+PMzx44sMGg4vWIHGAQfDGZ9FYRmrRT fLNprUrkUsIzVEDHcuYjrZfPHvqr4HK8KXa5IJIKj6WkCYj5gxkQlH3g9zEaWXlN Mfsqr6uioK34vKEgQw9HvpDpMZaB5NpZ7RWr5vEP3G3AV120VNKIEqU09Pex/IJD oiOaroR7y1OZUaLuTSyut+4MGjHJ7k6TNr8H/CFdRfpLs7mP5tsJG42rzAhqTdyP 2D2m7sgpEN8ALkGd+IU3ihLOVQCrSq2T0jpr6tFhDe5uTenP3fXBeGgu8ynRzPFD kkpyZhT094UWKoq8n+IcHQrIcba4j6X/jDJt7SDx8IwfZaPoMpcBGQH1//Q+lTuC q5qM7olI5OzcSr4AhDld7OOjqO0pLXRfH5y8hy8mgWEiGgomK+N5OFTqXrQsuB/g BopfBpqHRRYVpI1VkAutmTpyNjm3dQjrHmRWXX2sFe9lxExpWo9VX4By9ytv9EXQ UFfSBF9TVe+EkZ4XxO+JMCcahNUsBh45rRXY3wVpH1qPwsOVp8FxYcOmuzQSTha9 mgOxWbRfqkQ=
=jrll
-----END PGP SIGNATURE-----

#241 lore raw

Re: reiser4 plugins

Quoted 11 lines · David Masover
Quoted 9 lines · [email protected] wrote:
Quoted 3 lines · David Masover

"Plugins" is a bad word. This user's combination of plugins is most likely identical to other users', it's just which ones are enabled, and which aren't? If they are all included, I assume they play nice.

Which ones are enabled. Exactly.

Reiser4 plugins are not per user, but per kernel. They are compiled in. The model is intended to ease the development process, nothing more. Apologies if the naming suggests more.

Hans

#242 lore raw

Re: reiser4 plugins

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 14 lines · Hans Reiser
Quoted 11 lines · David Masover
Quoted 6 lines · [email protected] wrote:
Quoted 3 lines · David Masover

"Plugins" is a bad word. This user's combination of plugins is most likely identical to other users', it's just which ones are enabled, and which aren't? If they are all included, I assume they play nice.

Quoted 6 lines
Quoted 1 line
Quoted 1 line

Which ones are enabled. Exactly.

Reiser4 plugins are not per user, but per kernel. They are compiled in. The model is intended to ease the development process, nothing more. Apologies if the naming suggests more.

But, to avoid confusion, the inclusion of a crytocompress plugin in a given kernel doesn't mean that all files accessed from that kernel are encrypted and compressed. It just means that you can pick an individual file and set it to be transparently encrypted/compressed.

That is what I meant by "enabled". Not per-user, but per-file.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr9FcHgHNmZLgCUhAQJ2+BAAgnqAcEg41VXTUmpUUZWMduWeJKKYDE16 QvWore6MK/FWoU8hqFMddTzJVpzKNH8NoZ1Jfm2nEC51nndckwtBDsuIlMb1CGC+ I+wCrpe1kjXTz8sTs++CwDg29YojHP1/Wa5MwnUgTEBkZxacPI7r0VXyt1oO+f/N Eu6Jpgl4IN3ba52/PS+dXCQrmOJDhLeHAb+l4rtRsK+LASJitqmGmpf4PRwEejM8 dWZyjEYdiAcMx6mkeB/lho5IIcp1Xi5QBACf6S8SHXvHxRW50cKeouAISR25Ttk2 Oa1vKhWPGo4IBREjK7fzgj6GwfpIBnUfE25ZRjrBRupWumwekaCAY91JOoRkdRI+ Lw70OZhqqIKQ/O46xqhyCrroP/6is5ZxLyIRe1q3qsQfsWUfHBOONRUBdtaTQlaa 3OWzAU49cxn4Jv9S9UEYEyKx9ggqJ5hd94wZpXPq4GogWt4S8cYK3d4ZtRIyXEBr qOrLoJoTF9WeT254u+uq5gLP5kxq7Z7J51aMXbGF+4luuGJPq/50Y4hbapAMQWFA v0Z8YNWoOnOJgYji/+u8qJhsfzBdi/dmWlhhy9Te1e1b/1+zHcsbCslgsIrG2spk 3uF1GOv5NorO2n7RhierhkUrkUsLLlFqf0vPmgMqJ9DG4h6wl3bOJkYShBKTYQB+ ldYcxERKMZ4=
=jv9S
-----END PGP SIGNATURE-----

#243 lore raw

Re: reiser4 plugins

Quoted 6 lines · David Masover

But, to avoid confusion, the inclusion of a crytocompress plugin in a given kernel doesn't mean that all files accessed from that kernel are encrypted and compressed. It just means that you can pick an individual file and set it to be transparently encrypted/compressed.

That is what I meant by "enabled". Not per-user, but per-file.

Doing key management in a secure manner is going to be *fun*. :)

#244 lore raw

Re: reiser4 plugins

In reply to David Masover
Quoted 25 lines · David Masover
Quoted 11 lines · Hans Reiser
Quoted 8 lines · David Masover
Quoted 5 lines · [email protected] wrote:
Quoted 2 lines · David Masover

"Plugins" is a bad word. This user's combination of plugins is most likely identical to other users', it's just which ones are

enabled, and

Quoted 9 lines
Quoted 4 lines
Quoted 4 lines
Quoted 1 line

which aren't? If they are all included, I assume they play nice.

Which ones are enabled. Exactly.

Reiser4 plugins are not per user, but per kernel. They are compiled in. The model is intended to ease the development process, nothing more. Apologies if the naming suggests more.

Quoted 7 lines

But, to avoid confusion, the inclusion of a crytocompress plugin in a given kernel doesn't mean that all files accessed from that kernel are encrypted and compressed. It just means that you can pick an individual file and set it to be transparently encrypted/compressed.

That is what I meant by "enabled". Not per-user, but per-file.

You are correct.

#245 lore raw

Re: reiser4 plugins

In reply to David Masover
Quoted 1 line · David Masover

Hans Reiser wrote:

[...]

Quoted 3 lines
Quoted 3 lines

Reiser4 plugins are not per user, but per kernel. They are compiled in. The model is intended to ease the development process, nothing more. Apologies if the naming suggests more.

What do you gain by this? It is just a kernel configuration option of sorts? Just name mangling of existing mechanisms for no good reason at all?

Quoted 6 lines

But, to avoid confusion, the inclusion of a crytocompress plugin in a given kernel doesn't mean that all files accessed from that kernel are encrypted and compressed. It just means that you can pick an individual file and set it to be transparently encrypted/compressed.

That is what I meant by "enabled". Not per-user, but per-file.

Wonderful! I carefully "transparently encrypt" my secret files, so /everybody/ can read them! Now /that/ is progress!

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#246 lore raw

Re: reiser4 plugins

Quoted 2 lines · Horst von Brand

Wonderful! I carefully "transparently encrypt" my secret files, so /everybody/ can read them! Now /that/ is progress!

All of this side feature argument is completely offtopic for the inclusion of reiser4, but oh well.

In any case, the real use for encrypted files (vs encrypted partitions) would be for doing things like tying keying into the login process so that your files are only accessible while you are logged in. This would be a very nice feature on a multiuser system.

#247 lore raw

Re: reiser4 plugins

In reply to Horst von Brand

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 8 lines · Horst von Brand
Quoted 1 line · David Masover

Hans Reiser wrote:

[...]

Quoted 7 lines
Quoted 3 lines
Quoted 3 lines

Reiser4 plugins are not per user, but per kernel. They are compiled in. The model is intended to ease the development process, nothing more. Apologies if the naming suggests more.

What do you gain by this? It is just a kernel configuration option of sorts? Just name mangling of existing mechanisms for no good reason at all?

I don't know why it's named "plugins", but the mechanism itself gains a lot. For one thing, certain new "plugins" for new kinds of files (cryptocompressed files, for instance) can be almost as easy as new plugins for (say) XMMS.

Still, Hans, it's high time to either rename these "plugins" or start talking about bytecode engines...

Quoted 10 lines
Quoted 6 lines

But, to avoid confusion, the inclusion of a crytocompress plugin in a given kernel doesn't mean that all files accessed from that kernel are encrypted and compressed. It just means that you can pick an individual file and set it to be transparently encrypted/compressed.

That is what I meant by "enabled". Not per-user, but per-file.

Wonderful! I carefully "transparently encrypt" my secret files, so /everybody/ can read them! Now /that/ is progress!

Explain to me how this is worse than how you currently encrypt your secret files?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr+bCHgHNmZLgCUhAQKnag/8DF1PjfQwEYaJWUOOY0roqn6b+/L8X/X1 kgFE3SyPJkNRuY6X/J3EQUht5v/EBEbk2ZE/oTWB+HGEeUv1juUmG7z2rWCvi8sY tXPG28iqOAjwvZa+2aq8o4ptW0+gaXVIbrNGEaWTXPgbxdxtoVMOAakBJl3FUchK pwZWJD+msuQuPFc42zQ3Zdvp+4/NhhC6qcSpZ9DGXHO9uDJDDdl3Xgf9+qOD+W7Y Bz4sGfNphIbJjydHs0DdcZPh49MhS+qd48Qw1WFskalF9dhZhHFq3MKIBFbg8bA8 MX9yY5XHrNPJqJXF96YtcJV9nZcXh4qEz81/JC7V+VYm/BME20a0jtUt5bMtGRjj kC8FxzVvamOmeXnzRnU2DKIlGlfTUmK1u6wQpMsoiwPZr9WH7WRxiP/tHPVYQbBc gPpg7POOz3M6Q3guZMGTxeYu7kHJm8s/zJVwXm04fz+03hrkab1q/w4cT0SaX4Pv naipO7nkdr5qoRzF2sBSZAsnauxGa0ViR0TP715FN+oX0ZRiSQcgEGc+zXz6qPb5 bSz29jG1iGkrepB6LU88gMZiyxc2qK1SHpZlKjn9iPKqpxpwEtpQToUBhRsAlFBv 9Z6KxrQYrUYfJ/Z9ktIUuvsvN3zbnnc7tmZGYNisqm03dq61kguAwOV66INSlYDj gFaw23Z9X58=
=xsgf
-----END PGP SIGNATURE-----

#248 lore raw

Re: reiser4 plugins

In reply to David Masover
Quoted 9 lines · David Masover
Quoted 1 line

Right. So please explain what crypto/raw/foo and crypto/inflated/foo.gz give you.

In that example (shouldn't have used the name "crypto", but oh well), it should be crypto/raw/foo.gz and crypto/inflated/foo -- where foo.gz is the gzip'ed file and foo is the transparently compressed/decompressed file. Basically, these are equivalent:

$ zcat crypto/raw/foo.gz
$ cat crypto/inflated/foo

I'm *quite* aware of what your preconceived notions think it *should* be.

Maybe the two examples I asked for have *real-world* meanings that you should be allowing for. Like, for instance, on a mail server, where the A/V software may need to unzip a file 5 or 6 times to find out if there's malicious content.

Or seeing if it's a ".zip bomb", where a small .zip will decompress to gigabytes.

Or I'm testing a new compression algorithm, to see if multiple compressions help (yes, I know that it *shouldn't* help - but I've seen real-world cases where the algorithm could only look at a 4K or 8K window at a time, and if you hit a *very* long run of duplicate 4K segments, a second compression would compress all the identical or near-identical "this is a 4K chunk" tokens...)

Quoted 7 lines
Quoted 3 lines

It's got a *LOT* to do with it if I created a *DIRECTORY*, to use *AS A DIRECTORY*, the way Unix-style systems have done for 3 decades, and suddenly my system is running like a pig because the kernel decided that it's a .zip file.

The kernel does not decide that. You do. And it doesn't automatically decide that every time you create a file. You have to use some interface to trigger the plugins.

Oh, I'm waiting for the fun the first time somebody deploys a plugin that has similar semantics to 'chmod g+s dirname/' ;)

Quoted 2 lines

I guess I need a new name for this approach. That's three possible ways of doing this?

I *said* you need to think this through in detail, didn't I? ;)

Quoted 5 lines

I remember discussing that, actually. It wouldn't automatically do this if you didn't want it to, but it would be nice if, say, it was something truly seekable like linux-2.6.12.zip, and linux-2.6.12 was a user-created symlink to linux-2.6.12.zip/.../contents, and we had a nice caching system...

I think you're highly deluded as to just how much or little performance gain this will get you. Model what happens with a kernel 'make' on a 256M machine with and without all that zipping and compressing, and assume that a constant 48M is available for caching of the linux-2.6.12/ tree.

Quoted 3 lines

This is nice because then you get exactly the same performance during "make" as you would with "unzip && make", only better, because files you don't ever use (lots of arch, for instance) are not unpacked.

Go read http://www.tux.org/lkml/#s7-7 and ponder until enlightenment arrives.

#249 lore raw

Re: reiser4 plugins

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 57 lines · [email protected] wrote:
Quoted 9 lines · David Masover
Quoted 1 line

Right. So please explain what crypto/raw/foo and crypto/inflated/foo.gz give you.

In that example (shouldn't have used the name "crypto", but oh well), it should be crypto/raw/foo.gz and crypto/inflated/foo -- where foo.gz is the gzip'ed file and foo is the transparently compressed/decompressed file. Basically, these are equivalent:

$ zcat crypto/raw/foo.gz
$ cat crypto/inflated/foo

I'm *quite* aware of what your preconceived notions think it *should* be.

Maybe the two examples I asked for have *real-world* meanings that you should be allowing for. Like, for instance, on a mail server, where the A/V software may need to unzip a file 5 or 6 times to find out if there's malicious content.

Or seeing if it's a ".zip bomb", where a small .zip will decompress to gigabytes.

Or I'm testing a new compression algorithm, to see if multiple compressions help (yes, I know that it *shouldn't* help - but I've seen real-world cases where the algorithm could only look at a 4K or 8K window at a time, and if you hit a *very* long run of duplicate 4K segments, a second compression would compress all the identical or near-identical "this is a 4K chunk" tokens...)

Quoted 7 lines
Quoted 3 lines

It's got a *LOT* to do with it if I created a *DIRECTORY*, to use *AS A DIRECTORY*, the way Unix-style systems have done for 3 decades, and suddenly my system is running like a pig because the kernel decided that it's a .zip file.

The kernel does not decide that. You do. And it doesn't automatically decide that every time you create a file. You have to use some interface to trigger the plugins.

Oh, I'm waiting for the fun the first time somebody deploys a plugin that has similar semantics to 'chmod g+s dirname/' ;)

Quoted 2 lines

I guess I need a new name for this approach. That's three possible ways of doing this?

I *said* you need to think this through in detail, didn't I? ;)

Quoted 5 lines

I remember discussing that, actually. It wouldn't automatically do this if you didn't want it to, but it would be nice if, say, it was something truly seekable like linux-2.6.12.zip, and linux-2.6.12 was a user-created symlink to linux-2.6.12.zip/.../contents, and we had a nice caching system...

Quoted 4 lines

I think you're highly deluded as to just how much or little performance gain this will get you. Model what happens with a kernel 'make' on a 256M machine with and without all that zipping and compressing, and assume that a constant 48M is available for caching of the linux-2.6.12/ tree.

Ignoring Hans' point, there is still a performance gain.

Assume we can do on-disk caching, similar to fscache/cachefs for nfs. Now, benchmark:

$ unzip linux-2.6.12.zip && make -C linux-2.6.12

versus the hypothetical

$ make -C linux-2.6.12.zip/.../contents

This is an automatic performance gain, in theory, because the second command is identical to unzipping just the parts you need into linux-2.6.12, then running "make". The one disadvantage is that because the unzipping is done on demand, it only really performs well if you can keep the "zip" binary cached. Even on most embedded systems, 54K of RAM is really not much to ask these days.

Also, once you've run "make" once, you get to run it as many times as you like, and so long as the on-disk cache of the zipfile is still there and valid, you never have the overhead of unzipping again.

Of course, this probably saves only a minute or two at most per kernel compile. But that doesn't mean there aren't real-world situations where this kind of architecture would be much more beneficial.

Quoted 6 lines
Quoted 3 lines

This is nice because then you get exactly the same performance during "make" as you would with "unzip && make", only better, because files you don't ever use (lots of arch, for instance) are not unpacked.

Go read http://www.tux.org/lkml/#s7-7 and ponder until enlightenment arrives.

So what? I don't intend to convince anyone based on how much slower/faster their kernel compiles. It's meant to illustrate the principle of the thing.

Besides, your point was that you could not run make inside of a kernel tarball/zipfile. Nobody ever suggested that you would actually want to. -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr9mfHgHNmZLgCUhAQJi0g//RGxFXWi8Om4EnKsZHcI0J7X3G6T9pj2a nwkWwjLdyg6jykdt3a5MTELBgOM2uT87k7CeO7TasA/T1ZkZ/y2Yw7x50YIgrb7j W1u5N/vfDLw3C2Nd6O2fe/b4ygReyBB6HQtNTw/k+XxDswxtEp0mcZHpNxt+W9B4 s0naezawRjF51P4ISqa6HoRo7vZUkXv+3CwuZinC5m8KnW2Us8Ww5uDjtNBLpJGR zs79w24zaj6VSHjF8lO6CuMKQLjSelMDXKDEkFHaybJgz7AckkcZg5c6VDTrc3/t m8HM5oyHWfRqVeQt9cMdLdcBZnhdbspSwQaNQkdkrZx1IX96mPoMNDUwk1B/TIi7 ++iqpkDYeOdg+DWzGPVpwQ+5LQC+7m8vRHv5dROIM6T89TnlUck2clBiPovzSP+8 KMR1R6F7qBPxJMkPcy2MNOVMkjN+VLSOCzOeOXVEUkNYdXjrJB5h3XIN5MyRR7C/ pRmjB2crzPTUz2yBatP+QUFNUMadikMqj44sEc/ie8iHbo9pfxQC/wY4M4VkJcf2 03spe2e8M+k3txj63O9TmJYgobkjx+d+cJ5Zm7DbKJmGlVaAGqmCXeNjxhTtBSwU yP2Jrz6Awu+nDxFxMAsN4GP17/0/aXdBhIYLPGyAJ9/HV7KHENIqIjnvD4e3bXnU shBrM+G1N5E=
=BXCi
-----END PGP SIGNATURE-----

#250 lore raw

Re: reiser4 plugins

Quoted 1 line · David Masover
$ make -C linux-2.6.12.zip/.../contents

I've yet to see how such automatic unzipping does not inherently introduce system insecurity into _existing_ applications by changing POSIX and UNIX semantics.

When I do this in my suid perl script:

open my $fh, '<', $ARGV[0];

I do _not_ mean this:
system '/bin/zip', 'x', $path1, $path2;

Neither do I want the kernel to unzip it, because that just introduces the kernel to a whole series of normally application-level vulnerabilities. I've seen situations where a specially doctored tarball can expand to a result file over 1000x the size of the original. Likewise, there have been cases where crafted tarballs have taken advantage of buffer overflows.

Can you illustrate for me with precise, clear, and unambiguous arguments how this can avoid all possible pitfalls, especially in cases where it's likely to break existing working privileged apps? Such extended operations, including the automatic encryption/decryption and most other non-standard filesystem features (Basically the whole '...' directory), should probably be left out of any patch submitted for inclusion until they can be _proven_ (or at least thoroughly checked) not to have undesirable results.

Cheers,
Kyle Moffett

--
Somone asked me why I work on this free (http://www.fsf.org/philosophy/)
software stuff and not get a real job. Charles Shultz had the best answer:

"Why do musicians compose symphonies and poets write poems? They do it because life wouldn't have any meaning for them if they didn't. That's why I draw cartoons. It's my life."
-- Charles Shultz

#251 lore raw

Re: reiser4 plugins

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 11 lines · Kyle Moffett
Quoted 1 line · David Masover
$ make -C linux-2.6.12.zip/.../contents

I've yet to see how such automatic unzipping does not inherently introduce system insecurity into _existing_ applications by changing POSIX and UNIX semantics.

When I do this in my suid perl script:

Quoted 7 lines

open my $fh, '<', $ARGV[0];

I do _not_ mean this:
system '/bin/zip', 'x', $path1, $path2;

Neither do I want the kernel to unzip it, because that just introduces the kernel to a whole series of normally application-level vulnerabilities.

That just means the zip plugin has to be more carefully written than I thought. It would have to be sandboxed and limited to prevent buffer overruns and zip bombs.

I think that you could create a situation where an untrusted zip could be explored, and the worst side effect would be that the files you see inside the untrusted zip wouldn't be what you expect. But that's no surprise -- if the zip is really malicious, there's no amount of sandboxing that would give you valid files anyway.

I was probably taking this too lightly, though.

Remember that zip, at least, would not be in the kernel. I think what is going in the kernel is gzip and lzo, and it's being done so transparently that you never actually see the compressed version.

Quoted 1 line

Can you illustrate for me with precise, clear, and unambiguous arguments

That will take some time. And some thinking.

Quoted 1 line

how this can avoid all possible pitfalls,

Especially if you want perfection.

Quoted 5 lines

including the automatic encryption/decryption and most other non- standard filesystem features (Basically the whole '...' directory), should probably be left out of any patch submitted for inclusion until they can be _proven_
(or at least thoroughly checked) not to have undesirable results.

I am doing that out of habit. Actually, it probably ends up more as:
.../foo.zip/
instead of
foo.zip/.../

But, it is left out. At least that interface.

Now, the cryptocompress as it currently stands does not involve ... and does not introduce any new security holes in the way that you are describing. There might be some issues with key management (someone hinted vaguely at that), but nothing insurmountable.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQr9xZ3gHNmZLgCUhAQKmOA/9EGus4fGXKnBPoK4Rpd9K/6tgy0A7QFIO EeF50BSgl7M5sot9Vp28Dg1DA9X8gXHf/BxWIUse2doEdrbYKy3HFNd4ZChPFXCS j4ZtJo/eGYQntIFZ+exNJG2gDeprSBUH9AnMxF9xBfG8CxBdl6WL1dx8d7kc7ip/ UYGiu+9WmC9LanEb0ezhsO8I0KBvjx73Q3FTbD9N0cMIzK1Drv7p9r9rhswsoIzg eZnKrT2Z0u8BASbd0GfCAT3DH3Zn6zHf6Zk9FOaPaqcLwWlXbELaTbFvBp+2rpnH 9j3NwKHTtCKX5Z2BOQqtiEDEE8QInuDlcEV2K/y4g9YM1mMw3TNoXswaZrbPWWeF RD/YyzknaDhfgQdz9kQ2bfHJM7//Y9IUJZp/3NmWhvhc2+QBhXBBoTb8UEnRl7tK D9UIgsDFDVHlLcO9KPKokjyf3fRL57dd0ictHFORvicVIK6NFSfFP9LY/ZsmUXF9 Fbqwwuu8/5n9z5j9IyIqf5m7XJkHcZCeLFDkn89VS4ZM8W1aB0g3yRIlhSEDDkVt 0nZZRzvIlRHCPoPMZuoFhWSngi50ZAYXlRicKrHQERP8f7ECkB1JvY2rLSFqM+FX BGUZpXxgHe/zgg806ACNbfdnny5WgZ7qXl/IXD9svQa3lIgxY6We8ACj8Oi6c7eu 5ooBcT7AmRE=
=pBA8
-----END PGP SIGNATURE-----

#252 lore raw

Re: reiser4 plugins

Quoted 9 lines · David Masover
Quoted 2 lines

Neither do I want the kernel to unzip it, because that just introduces the kernel to a whole series of normally application-level vulnerabilities.

That just means the zip plugin has to be more carefully written than I thought. It would have to be sandboxed and limited to prevent buffer overruns and zip bombs.

Quoted 8 lines

Remember that zip, at least, would not be in the kernel. I think what is going in the kernel is gzip and lzo, and it's being done so transparently that you never actually see the compressed version.

Quoted 1 line

Can you illustrate for me with precise, clear, and unambiguous arguments

That will take some time. And some thinking.

Precisely. Come back when you have a good sturdy set of arguments. See the FUSE project (Also discussed in this thread), for better ideas on how to add strange filesystem semantics. NOTE: Even the FUSE project, which is in _userspace_ (as opposed to the massive kernelspace mess of reiser4), is taking a lot of flak for changing UNIX semantics, so I see no reason why Reiser4 should be special.

Quoted 3 lines
Quoted 1 line

how this can avoid all possible pitfalls,

Especially if you want perfection.

It doesn't need perfection, it just needs to convince a large segment of the developers on this list (and especially linus).

Quoted 11 lines
Quoted 4 lines

including the automatic encryption/decryption and most other non- standard filesystem features (Basically the whole '...' directory), should probably be left out of any patch submitted for inclusion until they can be _proven_ (or at least thoroughly checked) not to have undesirable results.

I am doing that out of habit. Actually, it probably ends up more as:
.../foo.zip/
instead of
foo.zip/.../

But, it is left out. At least that interface.

Ok, good. That's one of the first issues. A _lot_ of applications would get themselves thoroughly confused at that '...' interface, not to mention a lot of sysadmins :-D.

Quoted 3 lines

Now, the cryptocompress as it currently stands does not involve ... and does not introduce any new security holes in the way that you are describing.

Good. I think that we can all agree that, in the event cryptocompress is included, it would be a nice feature to have in all filesystems. Therefore, we should attempt to come up with a clean interface with which it could be added _inside_ the VFS.

Quoted 2 lines

There might be some issues with key management (someone hinted vaguely at that), but nothing insurmountable.

Likewise, this should be handled in a common VFS interface that all FSs can use. There already exists a key management system that would not be particularly difficult to interface with, but it would take thought and discussion.

Cheers,
Kyle Moffett

--
There are two ways of constructing a software design. One way is to make it so simple that there are obviously no deficiencies. And the other way is to make it so complicated that there are no obvious deficiencies.
-- C.A.R. Hoare